globo.com Profiling em Python

Friday, October 4, 13

porque profiling é importante?

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

então vamos otimizar tudo!

Friday, October 4, 13

Friday, October 4, 13

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil” - Donald Knuth

Friday, October 4, 13

from timeit import timeit

if __name__ == "__main__": setup = "from htmlmin.minify import html_minify;" setup += "from data import raw_html" t = timeit( stmt="html_minify(raw_html)", setup=setup, number=100) print(t)

benchmark.py

Friday, October 4, 13

$ python benchmark.py

25.8121981621

Friday, October 4, 13

conheça seu código

Friday, October 4, 13

‣ cProfile

‣ Profile

‣ hotshot (deprecated)

‣ trace

‣ line profiler

‣ memory profiler

Friday, October 4, 13

from data import raw_htmlfrom htmlmin.minify import html_minify

if __name__ == "__main__": html_minify(raw_html)

profile.py

Friday, October 4, 13

$ python -m cProfile profile.py

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

$ python -m cProfile -o out profile.py

Friday, October 4, 13

$ python -m cProfile -o out profile.py

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

>>> import pstats>>> p = pstats.Stats("out")>>> p.sort_stats("cumulative").print_stats(10)

315165 function calls (311828 primitive calls) in 1.334 seconds

Ordered by: cumulative time List reduced from 675 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.092 0.092 1.358 1.358 profile.py:2(<module>) 1 0.038 0.038 0.879 0.879 /(...)/htmlmin/minify.py:7(<module>) 1 0.001 0.001 0.840 0.840 /(...)/bs4/__init__.py:17(<module>) 1 0.073 0.073 0.839 0.839 /(...)/bs4/builder/__init__.py:1(<module>) 1 0.027 0.027 0.511 0.511 /(...)/bs4/builder/_html5lib.py:2(<module>) 1 0.028 0.028 0.483 0.483 /(...)/html5lib/__init__.py:12(<module>) 1 0.001 0.001 0.387 0.387 /(...)/htmlmin/minify.py:26(html_minify) 2 0.000 0.000 0.297 0.148 /(...)/bs4/__init__.py:80(__init__) 2 0.000 0.000 0.296 0.148 /(...)/bs4/__init__.py:193(_feed) 2 0.000 0.000 0.296 0.148 /(...)/bs4/builder/_html5lib.py:33(feed)

Friday, October 4, 13

$ kernprof.py -l -v minify.py

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

$ python -m memory_profiler minify.py

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

gui?

Friday, October 4, 13

Friday, October 4, 13

Friday, October 4, 13

outras ferramentas‣ meliae

‣ heapy (guppy)

‣ benchy

‣ valgrind

‣ python object graphs (objgraph)

‣ plop

‣ pycounters

Friday, October 4, 13

bônus

Friday, October 4, 13

django profiling

Friday, October 4, 13

algumas ferramentas‣ django-debug-toolbar

‣ django-profiler

‣ new relic

Friday, October 4, 13

Friday, October 4, 13

$ newrelic-admin run-program gunicorn -w 3 wsgi:application

Friday, October 4, 13

Friday, October 4, 13

globo.com Estamos contratando!

Friday, October 4, 13

globo.com obrigada!@flaviamissi

Friday, October 4, 13