llvm-py: Writing Compilers In Python

llvm-py: Writing Compilers Using PythonPyCon India 2010

Mahadevan Rmdevan@mdevan.org

September 25, 2010

Mahadevan R llvm-py: Writing Compilers Using Python September 25, 2010 1 / 14

About This Talk

Outline

Compilers overview, some terminology

LLVM about the “Low Level Virtual Machine”

llvm-py Python bindings for LLVM

About me:

Open source developer

Day job is as a software architect (medical imaging)

Started my career in 1999; mostly C, C++, a little Python!

..also the author of llvm-py.

Compilers What is a Compiler?

What is a Compiler?

What is a compiler?

Something that transforms “source code” into “something else”

What is “source code”?

A well-structured, textual representation of a programNot: preprocessors, assemblers

What is “something else?”

Assembly languageAnother well-structured textual representation (e.g. cfront)Intermediate, binary representation (AOT compilers)In-memory executable code (JIT compilers)Executable images

What is a Compiler?

What is a compiler?

What is a Compiler?

What is a compiler?

What is a Compiler?

What is a compiler?

A well-structured, textual representation of a program

Not: preprocessors, assemblers

What is a Compiler?

What is a compiler?

What is a Compiler?

What is a compiler?

What is a Compiler?

What is a compiler?

Assembly language

Another well-structured textual representation (e.g. cfront)Intermediate, binary representation (AOT compilers)In-memory executable code (JIT compilers)Executable images

What is a Compiler?

What is a compiler?

Assembly languageAnother well-structured textual representation (e.g. cfront)

Intermediate, binary representation (AOT compilers)In-memory executable code (JIT compilers)Executable images

What is a Compiler?

What is a compiler?

Assembly languageAnother well-structured textual representation (e.g. cfront)Intermediate, binary representation (AOT compilers)

In-memory executable code (JIT compilers)Executable images

What is a Compiler?

What is a compiler?

Assembly languageAnother well-structured textual representation (e.g. cfront)Intermediate, binary representation (AOT compilers)In-memory executable code (JIT compilers)

Executable images

What is a Compiler?

What is a compiler?

Compilers Inside a Compiler

Inside a Compiler

What does the compiler do with the source code?

Lexical analysis produces tokens

Parser eats tokens, produces AST

What is an AST? And what is abstract about it?

What next after an AST?

What is an intermediate form (IR)?

example: IRs in gcc: source → GENERIC → GIMPLE → RTL →backend

What is Static Single Assignment (SSA) form?

Inside a Compiler

Compilers Passes, Optimization and Code Generation

Passes, Optimization and Code Generation

What is a “pass”?

What is a “optimization”?

What is a “code generator”?

What is a “code generator generator”?

Compilers The Frontend and The Backend

The Frontend and The Backend

Everything before the first IR is usually called the frontend

And everything after as the backend

LLVM and llvm-py deal only with backends

Frontends tend to be simpler

Many frontends for a backend is common (Scala, Groovy, Clojure:Java; C#, IronPython, etc: .NET)

To build a front end in Python:

hand-coded lexer, (rec-desc) parserAntlr (generates Python)Yacc, Bison, Lemon (generates C, wrap to Python)Sphinx (C++, wrap to Python)various Python parser generators

hand-coded lexer, (rec-desc) parser

Antlr (generates Python)Yacc, Bison, Lemon (generates C, wrap to Python)Sphinx (C++, wrap to Python)various Python parser generators

hand-coded lexer, (rec-desc) parserAntlr (generates Python)

Yacc, Bison, Lemon (generates C, wrap to Python)Sphinx (C++, wrap to Python)various Python parser generators

hand-coded lexer, (rec-desc) parserAntlr (generates Python)Yacc, Bison, Lemon (generates C, wrap to Python)

Sphinx (C++, wrap to Python)various Python parser generators

hand-coded lexer, (rec-desc) parserAntlr (generates Python)Yacc, Bison, Lemon (generates C, wrap to Python)Sphinx (C++, wrap to Python)

various Python parser generators

LLVM What is LLVM?

What is LLVM?

Primarily a set of libraries..

..with which you can make a compiler backend..

..or a VM with JIT support (but hard to get this right)

It provides:

IR data structure, with text and binary representationswide range of optimizationscode generator (partly description-based) for many platformsJIT-compile-execute from IRlink-time optimization (LTO)support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

LLVM What is LLVM?

What is LLVM?

It provides:

LLVM What is LLVM?

What is LLVM?

It provides:

LLVM What is LLVM?

What is LLVM?

It provides:

IR data structure, with text and binary representations

wide range of optimizationscode generator (partly description-based) for many platformsJIT-compile-execute from IRlink-time optimization (LTO)support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

IR data structure, with text and binary representationswide range of optimizations

code generator (partly description-based) for many platformsJIT-compile-execute from IRlink-time optimization (LTO)support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

IR data structure, with text and binary representationswide range of optimizationscode generator (partly description-based) for many platforms

JIT-compile-execute from IRlink-time optimization (LTO)support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

IR data structure, with text and binary representationswide range of optimizationscode generator (partly description-based) for many platformsJIT-compile-execute from IR

link-time optimization (LTO)support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

IR data structure, with text and binary representationswide range of optimizationscode generator (partly description-based) for many platformsJIT-compile-execute from IRlink-time optimization (LTO)

support for accurate garbage collection

LLVM What is LLVM?

What is LLVM?

It provides:

LLVM LLVM Highlights

LLVM Highlights

Written in readable C++

Reasonable documentation, helpful and mature community

clang is a now-famous subproject of LLVM

llvm-gcc is gcc frontend (C, C++, Java, Ada, Fortran, ObjC) +LLVM backend

Many users: Unladen Swallow, Iced Tea, Rubinus, llvm-lua, GHC,LDC

LLVM Highlights

LLVM The LLVM IR

The LLVM IR

The IR looks like this (textual representation):

“Hello world” in LLVM Assembly

@msg = private constant [15 x i8] c"Hello , world !\0A\00"

declare i32 @puts(i8*)

define i32 @main(i32 %argc , i8** %argv) {

entry:

%0 = getelementptr [15 x i8]* @msg , i64 0, i64 0

%1 = tail call i32 @puts(i8* %0)

ret i32 undef

LLVM A Word About clang

A Word About clang

A sub-project of the LLVM team

clang is a C, Objective C and C++ frontend

works with the LLVM backend (obviously!)

It can convert C/ObjC/C++ code to LLVM IR

which can be played around with using llvm-py

and then compiled “normally”

A Word About clang

llvm-py What is llvm-py?

What is llvm-py?

Python bindings for LLVM

Including JIT

Excellent for experimenting and prototyping

Available in Ubuntu, Debian, MacPorts (but may be out of date)

Current version (0.6) works with LLVM 2.7

What is llvm-py?

Including JIT

What is llvm-py?

Including JIT

What is llvm-py?

Including JIT

What is llvm-py?

Including JIT

llvm-py More on llvm-py

More on llvm-py

Covers most of the LLVM APIs, esp. IR

See examples to get started

User guide at http://www.mdevan.org/userguide.html

Some LLVM knowledge required: Modules, IR instructions, ...

Learning curve of LLVM is steep, llvm-py/Python helps!

Users group: http://groups.google.com/group/llvm-py

More on llvm-py

llvm-py llvm-py Internals

llvm-py Internals

Wraps LLVM’s C API as a Python extension module

Extends LLVM C API as it is incomplete

Does not use binding generators (Boost.Python, swig etc)

Public APIs are in Python, use extension module internally

Mostly straightforward Python, a little metaclass magic

llvm-py Internals

Thanks References and Thanks

Thanks!

Thanks for listening!

Must read: Steven S. Muchnick. Advanced Compiler Design &Implementation.

Me: Mahadevan R, mdevan@mdevan.org, http://www.mdevan.org/

llvm-py: Writing Compilers In Python

Technology

Transcript of llvm-py: Writing Compilers In Python

Collective characterization, optimization and design of ... · ICC 10.1 ICC 11.0 ICC 11.1 ICC 12.0 ICC 12.1 LLVM 2.6 LLVM 2.7 LLVM 2.8 LLVM 2.9 LLVM 3.0 Phoenix MVS XLC Open64 Jikes

Llvm+Ghv Thesis

Compiling with Continuations and LLVMIntroduction LLVM The LLVM Landscape LLVM IR ARM64 x86-64 Power Compiler Optimizer LLVM Rust C SML Haskell Erlang PML e GHC VM MLton Rustc Clang

The LLVM Compiler Framework and Infrastructure 15-745: Optimizing Compilers David Koes 1/22/2008 Substantial portions courtesy Chris Lattner and Vikram.

LLVM Register Allocation

LLVM Overview

Attacking Client Side JIT Compilers · LLVM • An instruction set, a suite of libraries and a collection of tools designed around compilation. • A suite of libraries from the start

Advanced Compiler Design - GitHub PagesCourse project •Goal of the project –Get some hands on experience with compilers –Two options, most will do option 1 •Option 1: LLVM

PART I OVERVIEWG021 G149 50u 100u Py Py Sp Sp Py Py Cp Cp Co L024 100u 30u Py Gn Sp Goe Gn Py Py Py Aca 100u 10u Py-pyrite, Au-gold, Gn-galena, Cp-chalcopyrite Cha-chalcanthite, Bo

VRP in LLVM

lldb reproducers - LLVM

The Spack2Package2Manager:2 … · apex cram glib libcircle llvm-lld netlib-blas py-basemap py-pyelftools R tmux arpack cscope glm libdrm lmdb netlib ...

Autovectorization in llvm

LLVM Assembly Language Reference Manual - Arkanisarkanis.de/.../llvm-language-reference-manual.pdf · This document is a reference manual for the LLVM assembly language. LLVM is a

What is LLVM? And a Status Update.€¦ · Clang, LLVM, etc. LLVM/Clang is both a research platform and a production-quality compiler. LLVM is a liberally-licensed(*) infrastructure

Integrating LLVM with the PGI Fortran, C and C++ compilers · Integrating LLVM with the PGI Fortran, C and C++ compilers Third Workshop on the LLVM Compiler Infrastructure in HPC

CodeCompass - LLVM

CS153: Compilers Lecture 7: Structured Data in LLVM IR

Simple and Efficient SSA Construction - KIT · 2013-05-08 · Modern SSA-based compilers take diﬀerent approaches to construct SSA: For example, LLVM uses Cytron et al.’s algorithm

Introduction: a Toy Language · Compilers in a Heterogenous World Toy AST LLVM IR Machine IR Toy Asm TIR Shape Inference Function Specialization (“TreeTransform”) High-Level Language