Lua: Programming III Overview of Lexing& Parsing ... · 2/3/2017 · Overview of Lexing& Parsing...

Lua: Programming IIIOverview of Lexing & ParsingIntroduction to Lexical Analysis

CS F331 Programming LanguagesCSCE A331 Programming Language ConceptsLecture SlidesFriday, February 3, 2017

Glenn G. ChappellDepartment of Computer ScienceUniversity of Alaska [email protected]

© 2017 Glenn G. Chappell

ReviewLua: Programming I — Variables, Values, Expressions

Summary§ Main program is code at global scope.§ String literals use double quotes, single quotes, or double brackets

with a balanced number of equals signs separating each pair of brackets: "abc" and [===[abc]===] are the same.

§ Only values have types; variables are references to values.§ Multiple assignment.§ Function print does quick & dirty output. io.write is preferred.§ “..” operator does string concatenation, with automatic number-to-

string conversion.§ Type errors are flagged at runtime, when statement is executed.

3 Feb 2017 CS F331 / CSCE A331 Spring 2017

See prog1.lua.

2

ReviewLua: Programming I — Functions

Summary§ A function definition begins with the keyword function.§ Variables default to global—except parameters, loop counter.§ Newlines are irrelevant.§ Call functions as usual.§ First-class functions.§ Also use keyword function to create an unnamed function.


See prog1.lua.

3

ReviewLua: Programming I — Tables

Summary§ Maps/dictionaries, arrays, objects, classes implemented using a

single PL feature: table, a key-value structure implemented internally as a hash table.

§ Table literals use braces, entries separates by commas. Key-value pair is key in brackets, equals sign, value: { …, ["a"]=56, … }

§ Access values using braces for index syntax, as in C++/Java.§ Delete a key by setting associated value to nil.§ Can mix types of keys, values.§ If a key looks like an identifier, then we can use dot syntax:

t["abc"] and t.abc are the same.§ Can put functions in tables.§ Make an array by listing values in braces without keys. Indices start

at one. arr = { 7, "abc", fibo, 5.34 }§ Length of array arr: #arr


See prog1.lua.

4

ReviewLua: Programming II — Flow of Control

Summary§ if COND then STMTS end§ if COND then STMTS else STMTS end§ if COND then STMTS elseif COND then STMTS … end§ while COND do STMTS end§ for VAR=FIRST, LAST do STMTS end§ for VAR=FIRST, LAST, STEP do STMTS end§ break: as in C++, Java (there is no “continue”)§ Iterator-based for-in loop. Examples:

§ for k, v in pairs(TABLE) do STMTS end§ for k, v in ipairs(TABLE_AS_ARRAY) do STMTS end

We will eventually write our own iterators.§ Note: not-equal operator: “~=”.


Note!

See prog2.lua.

5

ReviewLua: Programming II — Modules [1/2]

Lua module: an importable file—the kind of thing we would make a header/source combination for in C++.

Using a module§ To import a module into other code, use the keyword require:

mymod = require "mymod"

§ Then use module members with the dot operator:

io.write(mymod.add6(15).."\n");

§ The module itself is in a file whose name is the module name plus the Lua suffix: “mymod.lua”.

3 Feb 2017 CS F331 / CSCE A331 Spring 2017 6

ReviewLua: Programming II — Modules [2/2]

Writing a module§ A module returns a table containing the module members.§ In the module source: initialize module object as an empty table:

local mymod = {}

§ Things we do not want to export are local.§ Things we want to export are module members.

function mymod.add6(n)return n+6

end

§ Return the module object at the end of the file.

return mymod


See mymod.lua, prog2.lua.

7

ReviewLua: Programming II — Metatables

A Lua table can have an associated metatable.

setmetatable(t, mt) -- Make mt the metatable for t

An attempt to access a nonexistent key in a table causes function __index in the metatable to be called.

function mt.__index(tbl, key)return mt[key]

end

Metatables can be used to do something like classes & objects.


See prog2.lua.

8

ReviewLua: Programming II — Colon Operator

aaa:bbb(x, y)aaa.bbb(aaa, x, y)

The above are identical.

When a Lua function lies in a table, it does not know this. In order to have the function act on the table (as if it were a member function of an object), we must pass the table as a parameter.

The colon operator offers a convenient way to do this.


See prog2.lua.

9

ReviewLua: Programming II — Closures

A closure is a function that carries with it (some portion of) the environment in which it was defined.

In Lua, when we return a function from a function, we get a closure.

A closure can form a simpler alternative to a traditional OO construction (class, objects), particularly when a class exists primarily to support a single member function.

Closures are found in a number of PLs. Since the 2011 standard, C++ has had closures, in the form of lambda functions.


See prog2.lua.

See closure.cpp.

10

Lua: Programming IIICoroutines [1/5]

A coroutine is a function that can give up control (we say “yield”) at any point, and then later be resumed. Coroutines allow for cooperative multitasking: switching back and forth between various pieces of code, with each piece voluntarily giving up control so that others can execute.

Coroutines are available in Lua through the standard module coroutine.



The coroutine module contains four functions of interest to us.

yieldA coroutine yields a value to its caller, calling coroutine.yield, passing the value to be yielded. A coroutine only does a return when it is completely finished; after that, it cannot be resumed.

create

Do not call a coroutine directly. Rather, pass the function to coroutine.create; the return value is a coroutine object,with type thread. A coroutine is dealt with only through this coroutine object.

resume

To run a coroutine, either starting it or resuming it after a yield, pass the coroutine object to coroutine.resume. This returns two values: an error code (discussed shortly) and the yielded value—nil if none.

statusPass the coroutine object to coroutine.status to get its state. This returns a string, which is "dead" if the coroutine has terminated.



To write a coroutine, write an ordinary Lua function.Each time there is a value to send back to the caller—or when

control should be given back to the caller—call coroutine.yield, passing the yielded value, if any.

When the coroutine has finished, return as usual.



To use a coroutine, first get the coroutine object, with coroutine.create.

cor = coroutine.create(cfunc)

Call coroutine.resume with the coroutine object to execute the coroutine. The first time resume is called, any additional parameters will be passed to the coroutine function. coroutine.resume returns an error code (true: okay, false: error) and the yielded value, if any.

If the value of coroutine.status is not "dead", then the coroutinewas successful, and any yielded value may be used. Then call coroutine.resume again to resume the coroutine and request another value, handling it as above.


See prog3.lua.


Some other PLs feature coroutines prominently.§ Go has the cutesily named “goroutines”.§ Python has long had coroutines with a simpler interface; they are

called “generators”. The yielded values are available to the caller via an iterator.

§ Recently, more general coroutines have been added to Python.§ C++ does not currently have coroutines per se, although equivalent

functionality can be implemented using the standard threads facility. Proposals for adding coroutines to the language have been advanced.


Lua: Programming IIICustom Iterators [1/5]

Recall that Lua has an iterator-based loop: the for-in construction.

for k, v in pairs(t) do -- t is a tableio.write("Key: "..k..", value: "..v.."\n")

end

pairs takes a table and returns an iterator, which for-in uses.

We can write our own iterators. To do this, we need to understand what code like the following does in Lua.

for u, v1, v2 in XYZ doFOR_LOOP_BODY

end



for u, v1, v2 in XYZ doFOR_LOOP_BODY

end

The above is translated to this:

local iter, state, u = XYZlocal v1, v2while true do

u, v1, v2 = iter(state, u)if u == nil then

breakendFOR_LOOP_BODY

end


“v1, v2” may be replaced with an arbitrary number of variables, or just one variable, or no variables at all.

state is there so that function iter can keep around any data that it wants. But if we make iter a closure, then we do not need this. We are still required to pass state around, but it can be nil.

u is the value of interest. Function iter must return it. The loop body should probably do something with it. Function iter must take uas its second parameter, but it can ignore this parameter.

17


Here is a simplified model for creating a Lua iterator.§ The thing we pass to the for-in construction (“XYZ” in my example)

returns a function (“iter”) and two other values, which can be nil.§ The returned function (“iter”) must take two parameters, but it

may ignore them.§ Function iter should return however many values we want at each

iteration of the loop.§ Signal the end of the loop by making the value returned by iter a

nil (or the first value it returns, if iter returns multiple values).



When we use this simplified model, a custom iterator will have the following form.

function XYZ(…)function iter(dummy1, dummy2)

if … thenreturn nil -- Iterator exhausted

end…return ??? -- Next value (or values)

endreturn iter, nil, nil

end



Here is an actual custom iterator, based on the simplified model.

-- count: iterator. Given a, b. Counts from a up to b.function count(a, b)

function iter(dummy1, dummy2)if a > b then

return nilendlocal save_a = aa = a+1return save_a

endreturn iter, nil, nil

end


Code that uses this iterator:

for i in count(2, 6) doio.write(i.." ")

endio.write("\n")

The above prints: 2 3 4 5 6

20

See prog3.lua.

Overview of Lexing & ParsingIntroduction

Consider what a compiler needs to do.1. Determine whether the given program is syntactically

correct, and, if so, find its structure.2. Determine all identifiers and what they refer to.3. Determine types and check that no typing rules are broken—if

compiling code in a statically typed PL.4. Generate code in the target language.

A course on compilers would cover all four of the above. Here, we look at item #1, which is called parsing.

Parsing is often broken into two phases.§ Lexical analysis§ Syntax analysis


Overview of Lexing & ParsingLexing

Lexical analysis, or lexing, means breaking up the input into words, which we call lexemes (or tokens). It takes a stream of characters as input and outputs a stream of lexemes, each generally identified as belonging to a particular category.

A code module that does lexical analysis is a lexical analyzer, or lexer. This generally involves computation at the level of regular grammars and finite state machines.


LexerCharacter

StreamLexemeStream

cout << ff(12.6); cout << ff(12.6);

id op id litop

punctop

22

Overview of Lexing & ParsingParsing

Syntax analysis, or parsing, takes a stream of lexemes as input, and, if this is syntactically correct, outputs a representation of its structure. A parser involves a higher level of computation than a lexer, at the level of CFGs; there is almost always a stack used somewhere.

The representation that a parser outputs might be a parse tree (concrete syntax tree). But a more common choice is an abstract syntax tree (AST). Such a tree leaves out things like punctuation, which only serve to guide the parser.


ParserLexemeStream

ASTor Error

cout << ff(12.6);

id op id litop

punctop

expr

binOp: <<

expr

id: cout funcCall

expr

id: ff

numLit: 12.6

exprDetails may vary,

depending on AST specification.

23

Overview of Lexing & ParsingLexing + Parsing [1/2]

Note that we have used the term “parsing”to mean two different things: the entireprocess, or just the second phase (syntaxanalysis). So “parsing” has both a general meaningand a specific meaning. In practice, this rarely leads to misunderstandings.


ParserLexemeStream

ASTor Error

cout << ff(12.6);

id op id litop

punctop

expr

binOp: <<

expr

id: cout funcCall

expr

id: ff

numLit: 12.6

expr

LexerCharacter

Streamcout << ff(12.6);

Parsing

24

Overview of Lexing & ParsingLexing + Parsing [2/2]

Parsing is not always separated into two phases. But such a separation has a number of advantages.§ It makes the code more modular.§ It simplifies lexical analysis, since the more complicated parsing

algorithms do not need to be involved in that phase.§ It makes a parser easier to write and more portable, since this code

is insulated from the outside world: character sets, files, checking for I/O errors on input, etc.

§ It simplifies the implementation of parser lookahead: checking one or more lexemes ahead of the current lexeme.


ParserLexemeStream

ASTor Error

LexerCharacter

Stream

Parsing

25

Introduction to Lexical AnalysisLexeme Categories [1/6]

A lexer reads a character stream and outputs a lexeme stream. Each lexeme is generally placed into a category. We look at some common categories.


LexerCharacter

StreamLexemeStream

cout << ff(12.6); cout << ff(12.6);

id op id litop

punctop

26


An identifier is a name a program gives to some entity: variable, function, type, namespace, etc.

In the C++ code below, the identifiers are circled.

class MyClass {public:

void myFunc(Type1 & aa, int bb) const{

for (int ii = -37; ii <= bb; ++ii){

aa.foo();}

}};



A keyword is an identifier-like lexeme that has special meaning within a programming language.

In the C++ code below, the keywords are circled.

class MyClass {public:

void myFunc(Type1 & aa, int bb) const{

for (int ii = -37; ii <= bb; ++ii){

aa.foo();}

}};



An operator is a word that gives an alternate method for making what is essentially a function call. The arguments of an operator are called operands.

Operators are often—but not always—placed between their operands; such an operator is an infix operator.

The arity of an operator is the number of operands it has. A unaryoperator has one operand. A binary operator has two operands.

In the following code, the operators are “+=” and “*”, which are binary operators, and “-”, which is a unary operator.

aaa += b * -c;



A literal is a bare value.

Here are some C++ literals.


Literal Kind of Literal-100.8f Numeric literal"Hello there!" Character array literal'x' Character literalfalse Boolean literal{ 1, 2, 3 } Initializer list literal

Since C++11

30


Punctuation is the category for the extra lexemes in a program that do not fit into any of the previously mentioned categories.

Punctuation in C++ includes braces ({ }), semicolons (;), an ampersand (&) indicating a reference, and a colon (:) after public or private.

Lexeme categories mentioned:§ Identifier§ Keyword§ Operator§ Literal§ Punctuation


Introduction to Lexical AnalysisReserved Words [1/3]

A reserved word is a word that fits the general specification of an identifier, but is not allowed as a legal identifier in a program. Note that, while this is an important concept, reserved word is not a lexeme category.

In many PLs, the keywords and the reserved words are the same.

However, it is not hard to envision a variant of (say) C in which the compiler could distinguish how a word is used, based on its position in the code. Then there could be keywords that are not reserved words. Something like the following might be legal.

for (for for = 10; for; --for) ;



Since the 2011 Standard, C++ has had two keywords that are not reserved words: override and final. These have special meaning when placed after the parentheses in a member-function declaration, but otherwise are simply identifiers.

So the following is legal C++.

class Derived : public Base {virtual void override() override;

// Derived member function named "override"// Overrides Base member function "override"

…



The programming language Fortran traditionally has no reserved words. The following is, famously, legal code in at least some versions of Fortran.

IF IF THEN THEN ELSE ELSE

On the other hand, there can be reserved words that are not keywords. The Java standard specifies that goto is a reserved word. However, it is not a keyword. Thus, this word cannot legally be included in a Java program at all.

Note: We will use the above definitions consistently in this class. Be aware, however, that the term keyword is sometimes used to mean what we mean by reserved word.


Introduction to Lexical AnalysisLexer Operation

There are essentially three ways to write a lexer.§ Automatically generated, based on regular grammars or regular

expressions for each lexeme category.§ Hand-coded state machine using a table.§ Entirely hand-coded state machine.

The first method might involve a software package like lex, which generates C code for a lexer, given input that consists mostly of regular expressions.

We will write a lexer using the last method. When we are done, it should not be difficult to see how we might have used a table instead.

A lexer outputs a series of lexemes. There is generally no need to store these lexemes in a data structure. Rather, the lexer can provide get-next-lexeme functionality, which the parser can then use.


Lua: Programming III Overview of Lexing& Parsing ... · 2/3/2017 · Overview of Lexing& Parsing...

Documents

Transcript of Lua: Programming III Overview of Lexing& Parsing ... · 2/3/2017 · Overview of Lexing& Parsing...