Getting deep in python features
Python Speleology
1. The Zen of Python
The Zen of Python
○ Beautiful is better than ugly.○ Explicit is better than implicit.○ Simple is better than complex.○ Complex is better than complicated.○ Flat is better than nested.○ Sparse is better than dense.○ Readability counts.○ Special cases aren't special enough to break the rules.○ Although practicality beats purity.○ Errors should never pass silently.○ Unless explicitly silenced.○ In the face of ambiguity, refuse the temptation to guess.○ There should be one -and preferably only one- obvious way to do it.○ Although that way may not be obvious at first unless you're Dutch.○ Now is better than never.○ Although never is often better than right now.○ If the implementation is hard to explain, it's a bad idea.○ If the implementation is easy to explain, it may be a good idea.○ Namespaces are one honking great idea -let's do more of those!
2. Types and objects
Types and objects: To be or not to be
a = 256
b = 256
print (a is b)
a = 257
b = 257
print (a is b)
Types and objects: To be or not to be
a = 256
b = 256
print (a is b)
a = 257
b = 257
print (a is b)
True
False
Types and objects: To be or not to be
a = 256
b = 256
print (a is b)
a = 257
b = 257
print (a is b)
True
False
a = 256
b = 256
print id(a), id(b)
a = 257
b = 257
print id(a), id(b)
22036112 22036112
22363568 22363640
Types and objects: To be or not to be
a = 256
b = 256
print (a is b)
a = 257
b = 257
print (a is b)
True
False
a = 256
b = 256
print id(a), id(b)
a = 257
b = 257
print id(a), id(b)
Types and objects: To be or not to be
a = 256
b = 256
print (a is b)
a = 257
b = 257
print (a is b)
True
False
a = 256
b = 256
print id(a), id(b)
a = 257
b = 257
print id(a), id(b)
22036112 22036112
22363568 22363640
Types and objects: functions are objects
def x(f):
return f
def y(f):
return x
print x(y)(8)(0)
Types and objects: functions are objects
def x(f):
return f
def y(f):
return x
print x(y)(8)(0)
0
Types and objects: functions are objects
def x(i):
if x.enabled:
return i
else:
return "disabled"
x.enabled = True
print x(8)
Types and objects: functions are objects
def x(i):
if x.enabled:
return i
else:
return "disabled"
x.enabled = True
print x(8)
What happened if not set x.enabled?
8
Types and objects: star operator
args = (3 , 6)
print range(*args)
args = { "name": "example" }
def f(name):
print name
f(**args)
def f(*args, **kwargs):
print args
print kwargs
f(1,2,3, name="example")
Types and objects: star operator
args = (3 , 6)
print range(*args)
args = { "name": "example" }
def f(name):
print name
f(**args)
def f(*args, **kwargs):
print args
print kwargs
f(1,2,3, name="example")
[3, 4, 5]
example
(1, 2, 3)
{'name': 'example'}
3. Reflection
Reflection: get & set
class Example(object):
num = 10
x = Example
dir(x)
hasattr(x, "num") == True
getattr(x, "num", 0) == 10
setattr(x, "num", 20)
Reflection: local & global
globals()
Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).
locals()
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
Reflection: __magic__
__name__
This is the name of the function. This only have a meaningful value is the function is defined with “def”.
__class__
This is a reference to the class a method belongs to.
__code__
This is a reference to the code object used in the implementation of python.
Reflection: inspector
from inspect import getcomments
# This is a comment
def f(x):
print x
print getcomments(f)
Reflection: inspector
from inspect import getcomments
# This is a comment
def f(x):
print x
print getcomments(f)
# This is a comment
Reflection: inspector
from inspect import getsource
# This is a comment
def f(x):
print x
print getsource(f)
Reflection: inspector
from inspect import getsource
# This is a comment
def f(x):
print x
print getsource(f)
def f(x):
print x
Reflection: let's more tricky
def f(x):
print x
print f.__code__.co_code
Reflection: let's more tricky
def f(x):
print x
print f.__code__.co_code
'd\x01\x00GHd\x00\x00S'
Reflection: let's more tricky
def f(x):
print x
print f.__code__.co_code
'd\x01\x00GHd\x00\x00S'
YEEES!!! THE BYTECODE!!!
4. Context manager
Context manager: in the beginning...
item = Item()
try:
item.open()
item.do()
finally:
item.close()
Context manager: nowadays...
with Item() as item:
item.do
class Item(object):
def __enter__(self):
self.open()
return self
def __exit__(self,exc_type,exc_value,exc_t):
self.close()
...
Context manager: the real world
with file("/tmp/test", "w") as f:
f.write("hello world")
with lock():
# do some concurrent
with sudo("root"):
# do some as root
5. Decorations
Decorations: bold & italic
def makebold(fn):
def wrapped():
return "<b>" + fn() + "</b>"
return wrapped
def makeitalic(fn):
def wrapped():
return "<i>" + fn() + "</i>"
return wrapped
@makebold
@makeitalic
def hello():
return "hello world"
print hello()
Decorations: bold & italic
def makebold(fn):
def wrapped():
return "<b>" + fn() + "</b>"
return wrapped
def makeitalic(fn):
def wrapped():
return "<i>" + fn() + "</i>"
return wrapped
@makebold
@makeitalic
def hello():
return "hello world"
print hello()
<b><i>hello world</i></b>
Decorations: complex decor
def makestyle(arg):
def decorator(f):
def wrapper(*args, **kw):
return "<" + arg + ">" + f() + "</" + arg + ">"
return wrapper
return decorator
@makestyle("b")
def hello():
return "hello world"
print hello()
Decorations: syntax sugar
def makebold(fn):
def wrapped():
return "<b>" + fn() + "</b>"
return wrapped
def makestyle(arg):
def decorator(f):
def wrapper(*args, **kw):
return "<" + arg + ">" + f() + "</" + arg + ">"
return wrapper
return decorator
makebold(hello)
makestyle("b")(hello)
6. Iterations
Iterations: comprehesions
squares = []
for x in range(10):
squares.append(x**2)
squares = [x**2 for x in range(10)]
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
{ (k,v) for k,v in [(1,2)] }
Iterations: comprehesions
squares = []
for x in range(10):
squares.append(x**2)
squares = [x**2 for x in range(10)]
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
{ (k,v) for k,v in [(1,2)] }
SET NOT DICT!
Iterations: comprehesions
squares = []
for x in range(10):
squares.append(x**2)
squares = [x**2 for x in range(10)]
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
{ (k,v) for k,v in [(1,2)] }
SET NOT DICT!
{ k:v for k,v in [(1,2)] }
Iterations: co-routine
def countdown(n):
print "Counting down from", n
while n > 0:
yield n
n -= 1
print "Done counting down"
for i in countdown(10):
print i
Iterations: co-routine
def countdown(n):
print "Counting down from", n
while n > 0:
yield n
n -= 1
print "Done counting down"
for i in countdown(10):
print i
Counting down from 10
10
9
8
7
6
5
4
3
2
1
Done counting down
Iterations: co-routine
def countdown(n):
print "Counting down from", n
while n > 0:
yield n
n -= 1
print "Done counting down"
print countdown(10)
<generator object at 0x4035874c>
7. Overloading
Overloading: binary operations
__add__(self, other) x + y
__sub__(self, other) x - y
__mul__(self, other) x * y
__div__(self, other) x / y
__pow__(self, other) x ** y
Overloading: binary operations
__radd__(self, other) y + x
__rsub__(self, other) y - x
__rmul__(self, other) y * x
__rdiv__(self, other) y / x
__rpow__(self, other) y ** x
Overloading: binary operations
__radd__(self, other) 1 + x
__rsub__(self, other) 1 - x
__rmul__(self, other) 1 * x
__rdiv__(self, other) 1 / x
__rpow__(self, other) 1 ** x
Overloading: binary operations
__iadd__(self, other) x+=y
__isub__(self, other) x-=y
__imul__(self, other) x*=y
__idiv__(self, other) x/=y
__ipow__(self, other) x**=y
Overloading: unary operations
__neg__(self) -x
__pos__(self) +x
__abs__(self) abs(x)
__invert__(self) ~x
Overloading: conversion operations
__int__(self) int(x)
__float__(self) float(x)
__complex__(self) complex(x)
__str__(self) str(x)
__nonzero__(self) bool(x)
__unicode__(self) unicode(x)
Overloading: comparison operations
__eq__(self, other) x == y
__lt__(self, other) x < y
__le__(self, other) x <= y
__gt__(self, other) x > y
__ge__(self, other) x >= y
__ne__(self, other) x != y
Overloading: containers
__contains__(self, other) y in x
__getitem__(self, other) x[y]
__setitem__(self, other,value) x[y] = z
__delitem__(self, other) del x[y]
__len__(self) len(x)
__reversed__(self) reversed(x)
__iter__(self) iter(x)
8. The Class Factory
The Class Factory: class & objects
class Example(object):
attribute = "this is a class attribute"
def __init__(self):
self.attribute = "this is an obj attr override class one"
self.another = "this is another obj attr, no class"
print Example.attribute
print Example().attribute
print Example().another
print Example.another
The Class Factory: class & objects
class Example(object):
attribute = "this is a class attribute"
def __init__(self):
self.attribute = "this is an obj attr override class one"
self.another = "this is another obj attr, no class"
print Example.attribute
print Example().attribute
print Example().another
print Example.another
this is a class attribute
this is an object attribute and override class one
this is another object attribute, no class
Traceback (most recent call last):
Line 11, in <module>
print Example.another
AttributeError: type object 'Example' has no attribute 'another'
The Class Factory: set & get
class Example(object):
def __init__(self):
self._name = x
@property
def name(self):
return self._name
@name.setter
def name(self, value):
self._name = value
The Class Factory: @property abuse
class Example(object):
def __init__(self, host, port):
self.host = host
self.port = port
@property
def connect(self):
lib.connect(self.host, self.port)
The Class Factory: @property abuse
class Example(object):
def __init__(self, host, port):
self.host = host
self.port = port
@property
def connect(self):
lib.connect(self.host, self.port)
NEVER TYPE METHODS AS PROPERTIES
The Class Factory: methods and more methods
@staticmethod
Nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.
@classmethod
Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).
The Class Factory: methods and more methods
@staticmethod
Nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.
@classmethod
Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).
@staticmethod
def static_method():
print "I do not receive nothing :("
@classmethod
def class_method(cls):
print "I'm a class %s" % str(cls)
The Class Factory: methods and more methods
class Example(object):
def __init__(self, name):
self.name = name
@classmethod
def class_method(cls, name):
return cls(name)
x = Example.class_method("example")
print x
The Class Factory: methods and more methods
class Example(object):
def __init__(self, name):
self.name = name
@classmethod
def class_method(cls, name):
return cls(name)
x = Example.class_method("example")
print x
<__main__.Example object at 0x40358b2c>
The Class Factory: children and parents
class Example(object):
def __init__(self, name):
self.name = name
def do_something(self):
raise NotImplementedError()
class ChildExample(Example):
def do_something(self):
print self.name
The Class Factory: children and parents
class ExampleA(object):
def __init__(self, name):
self.name = name
def do_something(self):
raise NotImplementedError()
class ExampleB(object):
def do_otherthing(self):
raise NotImplementedError()
class ChildExample(ExampleB, ExampleA):
def do_something(self):
print self.name
def do_otherthing(self):
print self.name
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
x = Example()
y = Example
print x
print y
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
x = Example()
y = Example
print x
print y
<__main__.Example object at 0x4035894c>
<class '__main__.Example'>
The Class Factory: τὰ μετὰ τὰ κλάση
def example():
class Example(object):
pass
return Example
x = example()
y = x()
print x
print y
The Class Factory: τὰ μετὰ τὰ κλάση
def example():
class Example(object):
pass
return Example
x = example()
y = x()
print x
print y
<class '__main__.Example'>
<__main__.Example object at 0x403589ec>
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
print type(Example)
print type(Example())
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
print type(Example)
print type(Example())
<type 'type'>
<class '__main__.Example'>
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
print type(Example)
print type(Example())
<type 'type'>
<class '__main__.Example'>
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
print type(Example)
print type(Example())
<type 'type'>
<class '__main__.Example'>
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
x = Example
y = type('Example', (object,), {})
print x
print y
print (x == y)
The Class Factory: τὰ μετὰ τὰ κλάση
class Example(object):
pass
x = Example
y = type('Example', (object,), {})
print x
print y
print (x == y)
<class '__main__.Example'>
<class '__main__.Example'>
False
The Class Factory: __magic__
__new__(cls, *args, **kwargs)
Is the first method to get called in an object's instantiation, is a @classmethod, and must return a new instance of type cls.
__init__(self, *args, **kwargs)
Is the initializer for the instance. It gets passed whatever the primary constructor was called with.
__del__(self)
Is the destructor of the instance, will be invoked before clean the reference to the instance.
The Class Factory: __metaclass__
def upper_attr(f_class_name, f_class_parents, f_class_attr):
attrs = ((name, value) \
for name, value in f_class_attr.items() \
if not name.startswith('__'))
uppercase_attr = dict((name.upper(), value) \
for name, value in attrs)
return type(f_class_name, f_class_parents, uppercase_attr)
class Foo(object):
__metaclass__ = upper_attr
bar = 'bip'
print hasattr(Foo, 'bar')
print hasattr(Foo, 'BAR')
The Class Factory: __metaclass__
def upper_attr(f_class_name, f_class_parents, f_class_attr):
attrs = ((name, value) \
for name, value in f_class_attr.items() \
if not name.startswith('__'))
uppercase_attr = dict((name.upper(), value) \
for name, value in attrs)
return type(f_class_name, f_class_parents, uppercase_attr)
class Foo(object):
__metaclass__ = upper_attr
bar = 'bip'
print hasattr(Foo, 'bar')
print hasattr(Foo, 'BAR')
False
True
9. Monkey Patching
Monkey patching: first try
class Person(object):
def speak(self):
print "hello"
def monkey(foo):
print "uh uh uh"
Person.speak = monkey
x = Person()
x.speak()
Monkey patching: be evil }:-)
import os
def monkey(*args, **kwargs):
print "no no no"
os.system = monkey
os.system("ls /tmp")
Monkey patching: be evil }:-)
import os
def monkey(*args, **kwargs):
print "no no no"
os.system = monkey
os.system("ls /tmp")
no no no
10. Profiling
Profiling: hand made
import time
class Timer(object):
def __init__(self, verbose=False):
self.verbose = verbose
def __enter__(self):
self.start = time.time()
return self
def __exit__(self, *args):
self.end = time.time()
self.secs = self.end - self.start
self.msecs = self.secs * 1000 # millisecs
if self.verbose:
print 'elapsed time: %f ms' % self.msecs
Profiling: hand made
import time
class Timer(object):
def __init__(self, verbose=False):
self.verbose = verbose
def __enter__(self):
self.start = time.time()
return self
def __exit__(self, *args):
self.end = time.time()
self.secs = self.end - self.start
self.msecs = self.secs * 1000 # millisecs
if self.verbose:
print 'elapsed time: %f ms' % self.msecs
from redis import Redis
rdb = Redis()
with Timer() as t:
rdb.lpush("foo", "bar")
print "=> elasped lpush: %s s" % t.
secs
with Timer as t:
rdb.lpop("foo")
print "=> elasped lpop: %s s" % t.secs
Profiling: profile & cProfile
try:
import cProfile as profile
except ImportError:
import profile
def fib(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)
def fib_seq(n):
seq = [ ]
if n > 0:
seq.extend(fib_seq(n-1))
seq.append(fib(n))
return seq
profile.run('print fib_seq(6); print')
Profiling: profile & cProfile
try:
import cProfile as profile
except ImportError:
import profile
def fib(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)
def fib_seq(n):
seq = [ ]
if n > 0:
seq.extend(fib_seq(n-1))
seq.append(fib(n))
return seq
profile.run('print fib_seq(6); print')
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765]
57356 function calls (66 primitive calls) in 0.746 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
21 0.000 0.000 0.000 0.000 :0(append)
20 0.000 0.000 0.000 0.000 :0(extend)
1 0.001 0.001 0.001 0.001 :0(setprofile)
1 0.000 0.000 0.744 0.744 <string>:1(<module>)
1 0.000 0.000 0.746 0.746 profile:0(print fib_seq(20); print)
0 0.000 0.000 profile:0(profiler)
57291/21 0.743 0.000 0.743 0.035 profile_fibonacci_raw.py:13(fib)
21/1 0.001 0.000 0.744 0.744 profile_fibonacci_raw.py:22(fib_seq)
11. Documentation
Documentation: Zen
Don't create documentation for your code.Code your documentation.
def elements(n):
"""Return a list of n numbers from 0 to n-
1.
"""
return range(0,n)
Documentation: everything is an object
def elements(n):
"""Return a list of n numbers from 0 to n-
1.
"""
return range(0,n)
print elements.__doc__
Documentation: everything is an object
def elements(n):
"""Return a list of n numbers from 0 to n-
1.
"""
return range(0,n)
print elements.__doc__
Return a list of n numbers from 0 to n-1
Documentation: style is important
def elements(n):
"""Return a list of n numbers from 0 to n-1.
:type n: int
:param n: the limit upper for the elements to be created (not
included).
:return: a class:`list` with the items.
"""
return range(0,n)
class Example(object):
"""This is the documentation of the class.
Usually you do not need it :)
"""
def __init__(self, param):
"""This is the documentation of the instance type.
"""
Documentation: do the work
$ sphinx-quickstart
Documentation: Bonus... abuse the doc
class Command(object):
"""Undocumented command
"""
class ListCommand(Command):
"""List command.
"""
def show_help(command):
cmd = getattr(globals()[__name__], "%sCommand" % command,
None)
if cmd is None:
return "Invalid command"
else:
return cmd.__doc__
print show_help("List")
12. Future
Future: pypy
Just in time compiler FAST!
With sandboxing
Best concurrency support
Future: python3
def hello(name: str, age: int) -> str:
return name
print hello.__anotations__
Future: python3
def hello(name: str, age: int) -> str:
return name
print hello.__anotations__
'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}{'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}
Future: python3
from functools import lru_cache
@lru_cache(maxsize=None)
def fib(n):
if n < 2:
return n
return fib(n-1)+fib(n-2)
Applauses & questionsNot necessarily in that order.
Top Related