AST: threats and opportunities
-
Upload
alexander-lifanov-ceo-ratingvc -
Category
Software
-
view
79 -
download
2
Transcript of AST: threats and opportunities
![Page 1: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/1.jpg)
ASTOpportunities and Threats
ITGM, Dec 2016
![Page 2: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/2.jpg)
RDD
TDD
Slap this sh*t & release
ITGM, Dec 2016
![Page 3: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/3.jpg)
Code Test
Test
Test
ITGM, Dec 2016
![Page 4: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/4.jpg)
Code Test
Test
Test
def f(a,b): return a+b
5 == f(2,3)
0 == f(2,-2)
1 == f(1,0)
ITGM, Dec 2016
![Page 5: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/5.jpg)
Code Test
Test
Test
def f(a,b): return a+b
5 == f(2,3)
0 == f(2,-2)
1 == f(1,0)
Test 1e15 == f(1,1e15-1)
ITGM, Dec 2016
![Page 6: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/6.jpg)
DIS
ITGM, Dec 2016
![Page 7: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/7.jpg)
>>> def foo(): ... a = 2 ... b = 3 ... return a + b ...
ITGM, Dec 2016
![Page 8: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/8.jpg)
>>> import dis >>> dis.dis(foo) 2 0 LOAD_CONST 1 (2) 3 STORE_FAST 0 (a)
3 6 LOAD_CONST 2 (3) 9 STORE_FAST 1 (b)
4 12 LOAD_FAST 0 (a) 15 LOAD_FAST 1 (b) 18 BINARY_ADD 19 RETURN_VALUE
ITGM, Dec 2016
![Page 9: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/9.jpg)
ITGM, Dec 2016
![Page 10: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/10.jpg)
![Page 11: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/11.jpg)
ITGM, Dec 2016
![Page 12: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/12.jpg)
![Page 13: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/13.jpg)
![Page 14: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/14.jpg)
Zephyr ASDLParser/Python.asdl
ITGM, Dec 2016
![Page 15: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/15.jpg)
Python/Python-ast.cimport _ast
ITGM, Dec 2016
![Page 16: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/16.jpg)
Lib/ast.pyimport ast
ITGM, Dec 2016
![Page 17: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/17.jpg)
• AST
• Add, And, Assert …
• NodeVisitor
• NodeTransformer
ITGM, Dec 2016
![Page 18: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/18.jpg)
class DepthVisitor(ast.NodeVisitor): depth = 0
def generic_visit(self, node): parent = node.parent if parent: while parent: self.depth += 1 parent = parent.parent ast.NodeVisitor.generic_visit(self, node)
ITGM, Dec 2016
![Page 19: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/19.jpg)
• parse
• copy_location
• fix_missing_locations
• dump
• increment_lineno
ITGM, Dec 2016
![Page 20: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/20.jpg)
Модификация AST
ITGM, Dec 2016
![Page 21: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/21.jpg)
class HackedImporter: def load_module(self, name): # do magic stuff return module
sys.path_hook.insert(0, HackedImporter)
ITGM, Dec 2016
![Page 22: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/22.jpg)
https://github.com/alifanov/ast-spbpython-example
ITGM, Dec 2016
![Page 23: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/23.jpg)
class ReturnIncrement(ast.NodeTransformer): def visit_Return(self, node): node.value = (ast.BinOp( left=node.value, op=ast.Add(), right=ast.Num(n=1) ) return node
ITGM, Dec 2016
![Page 24: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/24.jpg)
def transform_with(transformer): def transform_decorator(func): node = func_to_node(func) node = transformer().visit(node) node = ast.fix_missing_locations(node) func = node_to_func(node, func) return func return transform_decorator
ITGM, Dec 2016
![Page 25: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/25.jpg)
@transform_with(ReturnIncrementer) def f(a,b): return a+b
f(2,2) # 5
ITGM, Dec 2016
![Page 26: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/26.jpg)
Threats
• Debug glitch
• Errors with multiple applies
• Strange errors
• Compiler version dependence
ITGM, Dec 2016
![Page 27: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/27.jpg)
>>> parseprint("func(a, b=c, *d, **e)") # Python 3.4 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[Name(id='a', ctx=Load())], keywords=[keyword(arg='b', value=Name(id='c', ctx=Load()))], starargs=Name(id='d', ctx=Load()), # gone in 3.5 kwargs=Name(id='e', ctx=Load()))), # gone in 3.5 ])
>>> parseprint("func(a, b=c, *d, **e)") # Python 3.5 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[ Name(id='a', ctx=Load()), Starred(value=Name(id='d', ctx=Load()), ctx=Load()) # new in 3.5 ], keywords=[ keyword(arg='b', value=Name(id='c', ctx=Load())), keyword(arg=None, value=Name(id='e', ctx=Load())) # new in 3.5 ])) ])
ITGM, Dec 2016
![Page 28: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/28.jpg)
Opportunities• Constant propagation
• Autologger
• Autoprofiler
• Macros (karnickel)
• Extend language
• MacroPy
• PonyORM
• Linters (auto codereview)ITGM, Dec 2016
![Page 29: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/29.jpg)
>>> from macropy.tracing import macros, trace >>> trace[[x*2 for x in range(3)]] range(3) -> [0, 1, 2] x*2 -> 0 x*2 -> 2 x*2 -> 4 x*2 for x in range(3) -> [0, 2, 4] [0, 2, 4]
Macros
ITGM, Dec 2016
![Page 30: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/30.jpg)
welcome = gettext("Welcome, {}!").format(user_name)
welcome = gettext("Welcome, {}!".format(user_name))
Linters
ITGM, Dec 2016
![Page 31: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/31.jpg)
https://github.com/edx/edx-lint/blob/master/edx_lint/pylint/i18n_check.py
Linters
ITGM, Dec 2016
![Page 32: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/32.jpg)
TRANSLATION_FUNCTIONS = set([ '_', 'gettext', 'ngettext', 'ngettext_lazy', 'npgettext', 'npgettext_lazy', 'pgettext', 'pgettext_lazy', 'ugettext', 'ugettext_lazy', 'ugettext_noop', 'ungettext', 'ungettext_lazy', ])
def visit_callfunc(self, node): if not isinstance(node.func, astroid.Name): # It isn't a simple name, can't deduce what function it is. return
if node.func.name not in self.TRANSLATION_FUNCTIONS: # Not a function we care about. return
if not self.linter.is_message_enabled(self.MESSAGE_ID): return
first = node.args[0] if isinstance(first, astroid.Const): if isinstance(first.value, basestring): # The first argument is a constant string! All is well! return
# Bad! self.add_message(self.MESSAGE_ID, args=node.func.name, node=node)
ITGM, Dec 2016
![Page 33: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/33.jpg)
Tools• astdump
• astviewer
• ast_tool_box
• astroid
• baron
• redbaron
• astor
• meta
• astmonkey
ITGM, Dec 2016
![Page 34: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/34.jpg)
Recommendation unit tests system
ITGM, Dec 2016
![Page 35: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/35.jpg)
Pythoscope
ITGM, Dec 2016
![Page 36: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/36.jpg)
Smother
ITGM, Dec 2016
![Page 37: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/37.jpg)
app.views:add,app/tests.py::FunctionTest::test_add app.views:fact,app/tests.py::FunctionTest::test_fact app.views:hello,app/tests.py::FunctionTest::test_hello
ITGM, Dec 2016
![Page 38: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/38.jpg)
Code1 Test1
Code1 Test2
Code1 Test3
Code2 Test1
Code2 Test2
ITGM, Dec 2016
![Page 39: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/39.jpg)
def test_default_ordering(self): class OrderingListView(generics.ListAPIView): queryset = OrderingFilterModel.objects.all() serializer_class = OrderingFilterSerializer filter_backends = (filters.OrderingFilter,) ordering = ('title',) ordering_fields = ('text',)
view = OrderingListView.as_view() request = factory.get('') response = view(request) self.assertEqual( response.data, [ {'id': 3, 'title': 'xwv', 'text': 'cde'}, {'id': 2, 'title': 'yxw', 'text': 'bcd'}, {'id': 1, 'title': 'zyx', 'text': 'abc'}, ] ) # not in context
ITGM, Dec 2016
![Page 40: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/40.jpg)
def fact(a): if a == 0: return 1 return a * fact(a - 1)
def test_fact(self): self.assertTrue(fact(0) == 1) self.assertTrue(fact(1) == 1) self.assertTrue(fact(2) == 2) self.assertTrue(fact(3) == 6) self.assertTrue(fact(4) == 24)
ITGM, Dec 2016
![Page 42: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/42.jpg)
https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf
ITGM, Dec 2016
![Page 43: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/43.jpg)
ITGM, Dec 2016
![Page 44: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/44.jpg)
ITGM, Dec 2016
![Page 45: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/45.jpg)
ITGM, Dec 2016
![Page 46: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/46.jpg)
ITGM, Dec 2016
![Page 47: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/47.jpg)
ITGM, Dec 2016
![Page 48: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/48.jpg)
ITGM, Dec 2016
![Page 49: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/49.jpg)
Edges
ITGM, Dec 2016
![Page 50: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/50.jpg)
BinOpLeftNum
ITGM, Dec 2016
![Page 51: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/51.jpg)
Child Edge Parent Freq
Num left BinOp 12
Num value Assign 20
ITGM, Dec 2016
![Page 52: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/52.jpg)
Depth
ITGM, Dec 2016
![Page 53: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/53.jpg)
Builtin function
ITGM, Dec 2016
![Page 54: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/54.jpg)
![Page 55: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/55.jpg)
![Page 56: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/56.jpg)
https://github.com/alifanov/testedio
ITGM, Dec 2016
![Page 57: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/57.jpg)
Roadmap
ITGM, Dec 2016
![Page 58: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/58.jpg)
Improve searchmodules names as features
ITGM, Dec 2016
![Page 59: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/59.jpg)
Import github projectsunit-test monitoring
ITGM, Dec 2016
![Page 60: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/60.jpg)
Fuzzy recommendations
hypothesis*pylint plugin
ITGM, Dec 2016
![Page 61: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/61.jpg)
Performance optimizationmap -> list comprehension
*pylint plugin
ITGM, Dec 2016
![Page 62: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/62.jpg)
Other languagesJavaScript
ITGM, Dec 2016
![Page 63: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/63.jpg)
Real TDDtests -> code generation
ITGM, Dec 2016
![Page 64: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/64.jpg)
Linkshttps://github.com/mkwiatkowski/pythoscope http://smother.readthedocs.io/en/latest/index.html http://testmon.org/ https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf http://is.ifmo.ru/diploma-theses/2013/bachelor/rost/rost.pdf https://habrahabr.ru/post/65944/ http://pyke.sourceforge.net/
ITGM, Dec 2016
![Page 65: AST: threats and opportunities](https://reader031.fdocuments.in/reader031/viewer/2022021922/5872291f1a28ab3b7a8b5935/html5/thumbnails/65.jpg)
Contacts
Telegram: @jetbootsmaker
Alexander Lifanov
ITGM, Dec 2016