The Closures Controversy

Joshua BlochChief Java Architect

Google Inc.

www.javapolis.com

Disclaimer: Talk Represents My Opinion, Not Google’s!

Google believes that the Java platform would likely benefit from some additional support for closures. Like the broader Java community, we are divided on what form this support should take. Some favor a lightweight approach that addresses the pain of anonymous inner classes without affecting the type system, VM, or libraries; others favor a heavyweight approach designed to provide for programmer-defined control structures. We believe it is premature to launch a JSR that forces us down either path.

www.javapolis.com

Outline

I. Setting the Stage

II. Why Enhance Support for Closures?

III. BGGA Closures

IV. A Lightweight Approach

V. Where Do We Go From here?

OOPSLA Invited Talk

October 8, 1996

Digitally reconstructed

from the Internet Archive

(AKA the Wayback Machine)

by Joshua Bloch

November 24, 2007

Oak

• Started as a reimplementation of C++

• Always a tool, never an end itself

• Took on a life of its own

• The Web happened...• serendipitous match!

• and it became Java

The Java Language

Fusion of four kinds of programming

• Object Oriented like Simula/C++/ ObjectiveC…

• Numeric like FORTRAN

• Systems like C

• Distributed like nothing else

Java - a language for a job

• Doing language research:an anti-goal

• Started using C++

• Broke down, needed:• Architecture neutral, portable, reliable, safe,

long lived, multithreaded, dynamic, simple, ...

Practical, not theoretical

• Driven by what people needed• (but hey, I spent too much time going to school!)

• Theory provides• rigour• cleanliness• cohesiveness

No new ideas here

• Shamelessly ripped off ideas that worked in C, C++, Objective C, Cedar/Mesa, Modula, Simula, ...

• (well, we slipped once or twice and invented something)

Don’t fix it until it chafes

• To keep it simple...

• A procedural principle:• Require several real instances before

including a feature• i.e. nothing goes in because it’s “nice”

• (== “Just Say No, until threatened with bodily harm”)

Java feels...

• Hyped :-(

• Playful

• Flexible

• Deterministic

• Non-threatening

• Rich

• Like I can just write code…

Hey!

I

left

ou

t “O

bje

ct-

Ori

en

ted

”!

IEEE Computer, June 1997

www.javapolis.com

So How Are We Doing? Not So Well, Unfortunately

Enum<E extends Enum<E>> { ... }

<T extends Object & Comparable<? super T>> T Collections.max(Collection<? extends T>)

public <V extends Wrapper<? extends Comparable<T>>>

Comparator<V> comparator() { ... }

error: equalTo(Box<capture of ?>) in Box<capture of ?> cannot be applied to (Box<capture of ?>)

equal = unknownBox.equalTo(unknownBox)

Arrays.asList(String.class, Integer.class) // Warning!

See Angelia Langer's 427-page (!) Java Generics FAQ for more:

http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.pdf

www.javapolis.com

What the Man on the Web is Saying

“I am completely and totally humbled. Laid low. I realize now that I am simply not smart at all. I made the mistake of thinking that I could understand generics. I simply cannot. I just can't. This is really depressing. It is the first time that I've ever not been able to understand something related to computers, in any domain, anywhere, period.”

“I'm the lead architect here, have a PhD in physics, and have been working daily in Java for 10 years and know it pretty well. The other guy is a very senior enterprise developer (wrote an email system that sends 600 million emails/year with almost no maintenance). If we can't get [generics], it's highly unlikely that the ‘average’ developer will ever in our lifetimes be able to figure this stuff out.”

www.javapolis.com

Where Does The Complexity Come From?

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Features

Co

mp

lexi

ty

Features(linear)

Feature Pairs(quadratic)

Feature Tuples(exponential)

www.javapolis.com

If The Feel of Java is to be Preserved...

We simply cannot afford another wildcards Further language additions must be

undertaken with extreme caution Minimal addition to conceptual surface area High power-to-weight ratio

www.javapolis.com

Outline

I. Setting the Stage

II. Why Enhance Support for Closures?

II. BGGA Closures

III. A Lightweight Approach

IV. Where Do We Go From here?

www.javapolis.com

What is a Closure?

One definition: a function that is evaluated in an environment containing one or more bound variables [Wikipedia]

In English: a little snippet of code that can be passed around for subsequent execution

Limited support for closures since JDK 1.1, in the form of anonymous classes

www.javapolis.com

Why Are We Considering Better Support for Closures?

Fine-Grained Concurrecy - Passing snippets of code to fork-join frameworks using anonymous classes is a pain

Resource Managememnt - Using try-finally blocks for resource management is a pain, and causes resource leaks

www.javapolis.com

1. Fine-grained (fork-join) concurrency

"It has to be easier to send snippets of code to frameworks for parallel execution; otherwise no one will use them .“

–Doug Lea, 2005

www.javapolis.com

Here’s How it Looks Today

class StudentStatistics { ParallelArray<Student> students = ... // ... public double getMaxSeniorGpa() { return students.withFilter(isSenior). withMapping(gpaField).max(); }

// helpers: static final class IsSenior implements Predicate<Student> { public boolean evaluate(Student s) { return s.credits > 90; } } static final IsSenior isSenior = new IsSenior(); static final class GpaField implements MapperToDouble<Student> { public double map(Student s) { return s.gpa; } } static final GpaField gpaField = new GpaField(); }

www.javapolis.com

2. Automatic Resource Management

“The C++ destructor model is exactly the same as the Dispose pattern, except that it is far easier to use and a direct language feature and correct by default, instead of a coding pattern that is off by default and causing correctness or performance problems when it is forgotten.”

–Herb Sutter, OOPSLA 2004

www.javapolis.com

How it Looks Today–Manual Resource Management

static String readFirstLineFromFile(String path) throws IOException { BufferedReader r = null; String s; try { r = new BufferedReader(new FileReader(path)); s = r.readLine(); } finally { if (r != null) r.close(); } return s;}

www.javapolis.com

It’s Worse With Multiple Resources (Puzzler 41)

static void copy(String src, String dest) throws IOException { InputStream in = null; OutputStream out = null; try { in = new FileInputStream(src); out = new FileOutputStream(dest); byte[] buf = new byte[1024]; int n; while ((n = in.read(buf)) >= 0) out.write(buf, 0, n); } finally { closeIgnoringException(in); closeIgnoringException(out); }}

private static void closeIgnoringException(Closeable c) { if (c != null) { try { c.close(); } catch (IOException ex) { // ignore } }}

www.javapolis.com

Automatic Resource Management

C++ DestructorString^ ReadFirstLineFromFile( String^ path ) {

StreamReader r(path);

return r.ReadLine();

}

C# using BlockString ReadFirstLineFromFile( String path ) {

using ( StreamReader r = new StreamReader(path) ) {

return r.ReadLine();

}

}

www.javapolis.com

Outline

I. Setting the Stage

II. Why Enhance Support for Closures?

III. BGGA Closures

IV. A Lightweight Approach

V. Where Do We Go From here?

www.javapolis.com

Controversial Features in BGGA

Function types Non-local return Non-local break and continue Unrestricted access to nonfinal local variables Design goal: library-defined control constructs

www.javapolis.com

Function Types

The BGGA Spec says: “While the subtype rules for function types may at first glance appear arcane, they are defined this way for very good reason: [...]. And while the rules seem complex, function types do not add complexity to Java's type system because function types and their subtype relations can be understood as a straightforward application of generics and wildcards (existing constructs). From the programmer's perspective, function types just ‘do the right thing.’”

www.javapolis.com

Function Types are Hard to Read

static Pair<{ => int },{ => int }> joinedCounters(int initial) { return Pair.<{ => int },{ => int }>of( { => initial++ }, { => initial++ });}

interface BThunk extends {=>boolean} { }static final {BThunk, { => void} => void} wihle = {BThunk cond, { => void } action => while (cond.invoke()) action.invoke(); };

static <throws X> { {=> void throws X} => void throws X }foo() { return { { => void throws X } block => block.invoke(); };}

These examples come from test code that ships with BGGA Prototype

www.javapolis.com

Function Types Encourage an “Exotic” Programming Style

static <A1, A2, R> {A1 => {A2 => R}} curry({A1, A2 => R} fn) { return {A1 a1 => {A2 a2 => fn.invoke(a1, a2)}};} <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1); static <A1, A2, A3, R> {A2, A3 => R} partial({A1, A2, A3 => R} fn, A1 a1) { return {A2 a2, A3 a3 => fn.invoke(a1, a2, a3)};}

From Mark Mahieu's blog:Currying and Partial Application with Java Closureshttp://markmahieu.blogspot.com/2007/12/currying-and-partial-

application-with.html

www.javapolis.com

Nominal Types are Rich Compared to Function Types

{T, T => T}

Name →

Known Implementations →

Documentation, including semantic constraints

www.javapolis.com

Function Types Have Unexpected Interactions

Arrays don’t work Foo.java:6: generic array creation { => int}[] closures = new { => int}[N];

Autoboxing doesn’t work LoopBenchC.java:10: <E,X>forEach(java.util.Collection<E>,

{E => void throws X}) in LoopBenchC cannot be applied to (java.util.List<java.lang.Integer>,{int => void})

forEach(list, {int i =>

Wildcards produce difficult error messages NewtonWithClosures.java:26: invoke(capture#418 of ? super

{double => double}) in {capture#418 of ? super {double => double} => capture#928 of ? extends {double => double}} cannot be applied to (double)

return fixedPoint(transform.invoke(guess)); ^

www.javapolis.com

Function Types Limit Interoperability With SAM Types

Closure conversion only works with interfaces

Unfortunately, existing APIs sometimes use

abstract classes for functions e.g., TimerTask, SwingWorker

These APIs would become 2nd class citizens

www.javapolis.com

Summary - Pros and Cons of Function Types

+ Avoid need to define named interfaces+ Avoid incompatibility among SAM types with

same signatures- Hard to read under moderate-to-heavy use- Encourage “exotic” style of programming- Don't reflect semantic constraints- Don't provide the same level of documentation- Don’t interact well with autocompletion (or grep)- Limited interoperability may balkanize libraries

www.javapolis.com

BGGA Closures Have Two Kinds of Returns

static boolean test(boolean arg) { {boolean => boolean} closure = { boolean arg => if (arg) return true; // Non-local return false // local return }; return !closure.invoke(arg);}

return means something completely different in a BGGA closure and an anonymous class

www.javapolis.com

What Does test() Return?

static <E> Boolean contains(Iterable<E> seq, Predicate<E> pred) { for (E e : seq) if (pred.invoke(e)) return true; return false;}

static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, new Predicate<Character>() { public Boolean invoke(Character c) { return c > 'j'; } });}

interface Predicate<T> { Boolean invoke(T t); }

www.javapolis.com

Now What Does test() Return? (BGGA)

static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false;}

static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; } );}

www.javapolis.com

Now What Does test() Return? (BGGA)

static <E> Boolean contains(Iterable<E> seq, {E => Boolean} p) { for (E e : seq) if (p.invoke(e)) return true; return false;}

static Boolean test() { List<Character> list = Arrays.asList( 'h', 'e', 'l', 'l', 'o'); return contains(list, { Character c => return c > 'j'; });}

Accidental non-local return due to cut-and-paste from anonymous class can cause insidious bug

www.javapolis.com

Suppose You Wanted to Translate this Method to BGGA

static <E> Predicate<Iterable<E>> contains( final Predicate<E> pred) { return new Predicate<Iterable<E>>() { public Boolean invoke(Iterable<E> seq) { for (E e : seq) if (pred.invoke(e)) return true; return false; } };}

www.javapolis.com

It’s Awkward, as Only One Local Return is Permitted

static <E> { Iterable<E> => Boolean } contains( { E => Boolean } pred) { return { Iterable<E> seq => Boolean result = false; for (E e : seq) { if (pred.invoke(e)) { result = true; break; } } result };}

www.javapolis.com

Summary - Pros and Cons of Non-Local Returns

+ Permits library-defined control structures

- Having two kinds of returns is confusing

- Meaning of return has changed

- Unintentional non-local returns can cause bugs

- Only one local return permitted per closure

www.javapolis.com

What Does This Program Print?

public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>();

for (int i = 0; i < N; i++) closures.add( { => i } );

int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); }}

www.javapolis.com

What does this program print?

public class Test { private static final int N = 10; public static void main(String[] args) { List<{ => int}> closures = new ArrayList<{ => int}>();

for (int i = 0; i < N; i++) closures.add( { => i } );

int total = 0; for ({ => int} closure : closures) total += closure.invoke(); System.out.println(total); }}

It prints 100, not 45. The same loop variable is captured by all 10 closures and evaluated after the loop is finished!

www.javapolis.com

Summary - Pros and Cons of Access to Nonfinal Locals

+ Permits library-defined control structures

+ Eliminates some uses of final modifier

- Semantics can be very confusing- Locals can persist after their scope is finished

- Locals can be modified by other threads

- Performance model for locals will change

www.javapolis.com

Library-Defined Control Constructs

What are the compelling use-cases? Custom loops Automatic resource management blocks Timer block

www.javapolis.com

Custom for-loops (Example from BGGA Spec)

<K,V,throws X>void for eachEntry(Map<K,V> map, {K,V=>void throws X} block) throws X { for (Map.Entry<K,V> entry : map.entrySet()) { block.invoke(entry.getKey(), entry.getValue()); }}

for eachEntry(String name, Integer value : map) { if ("end".equals(name)) break; if (name.startsWith("com.sun.")) continue; System.out.println(name + ":" + value); }

www.javapolis.com

What's Wrong With This example?

BGGA loop competes with Java 5 for-each Don't define a construct; just implement Iterable

Only compelling use is multiple loop variables Last example doesn't offer power of for-each

Loop variable can’t be primitive (no auto-unboxing) Would require 81 (!) overloadings to do fix this

www.javapolis.com

Loop syntax tailored to for; awkward for while

public static <throws X> void for myWhile( {=> boolean throws X} cond, {=>void throws X} block)

throws X { while (cond.invoke()) { block.invoke(); }}

for myWhile( { => i < 7 } ) { System.out.println(i++); }

myWhile( { => i < 7 }, { => System.out.println(i++); });

www.javapolis.com

Automatic Resource Management Block (BGGA Spec)

<R, T extends Closeable, throws X>R with(T t, {T=>R throws E} block) throws X { try { return block.invoke(t); } finally { try { t.close(); } catch (IOException ex) {} }}

with (FileReader in : makeReader()) { // Requires nesting with (FileWriter out : makeWriter()) { // code using in and out } }

www.javapolis.com

Library-Defined Control Constructs are a Double-Edged Sword

A great feature of Java is “programmer portability” All Java code looks pretty much alike

We can read and maintain each other's code with a

minimum of effort

Library-Defined Control Constructs foster

dialects, which hinder programmer portability

www.javapolis.com

Performance is an Open Question

for-each BGGA % change

List 1.65 s 1.97 s +19%

Array 0.18 s 1.11 s +519%

Time to iterate over 108 elements(Take these numbers with a huge grain of salt)

Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)Windows XP, Intel T2600 @ 2.16 GHz, 2GB RAMBGGA prototype closures-2007-11-30 (I apologize)

www.javapolis.com

Summary - Programmer-Defined Control Constructs

+ They let you define new control constructs

- They don’t have the same syntax, semantics, or performance as built-in control constructs

- It’s not clear that you need the ability; Java already has a rich set of control structures

- Can lead to dialects

- Responsible for much of the complexity in BGGA- Non-local return, break and continue

- Unrestricted access to mutable local variables

www.javapolis.com

Outline

I. Setting the Stage

II. Why Enhance Support for Closures?

III. BGGA Closures

IV. A Lightweight Approach

V. Where Do We Go From here?

www.javapolis.com

A Lightweight Approach

There is a much simpler approach to reducing the verbosity of anonymous classes and manual resource management

Attack the two problems head-on! Concise syntax for anonymous class instance creation Purpose-built construct for automatic resource

management

www.javapolis.com

Concise Instance Creation Expressions (CICE)

This: sort(list, Comparator<String>(String s1, String s2) { return s1.length() - s2.length(); });

Expands to this: sort(list, new Comparator<String>() { public int compare(String s1, String s2) { return s1.length() - s2.length(); } });

I am not in love with this syntax; we can probably do better.

www.javapolis.com

Automatic Resource Management (ARM) Blocks

// One resource - readFirstLineFromFiletry (BufferedReader r = new BufferedReader(new FileReader(path)) { String s = r.readLine();}

// Multiple resources - copyFiletry (InputStream in = new FileInputStream(src); OutputStream out = new FileOutputStream(dest)) { byte[] buf = new byte[1024]; int n; while ((n = in.read(buf)) >= 0) out.write(buf, 0, n);}

www.javapolis.com

More Automatic Resource Management Blocks

Perhaps this: protected (lock) { ... // access the resource protected by this lock }

Should be shorthand for this: lock.lock(); try { // access the resource protected by this lock } finally { lock.unlock(); }

www.javapolis.com

For More Information

Automatic Resource Management Blocks (ARM) http://docs.google.com/View?docid=dffxznxr_1nmsqkz

Concise Instance Creation Expressions (CICE) http://docs.google.com/View?docid=k73_1ggr36h

Both of these docs are a bit sketchy

www.javapolis.com

Outline

I. Setting the Stage

II. Why Enhance Support for Closures?

III. BGGA Closures

IV. Lightweight Closures

V. Where Do We Go From here?

www.javapolis.com

The Closures Spectrum

Bob Lee

CICE+ARM

www.javapolis.com

The Big Questions

Do we really want function types in Java? Do we really want library-defined control

constructs in Java? We should keep in mind that there’s already

a fine programming language for the JVM that offers both of these things, and Java interoperability to boot: Scala

www.javapolis.com

The Medium-Sized Questions

Should we support non-local return, break, and continue? If so: Should it be the default? How do we prevent accidents?

Should we allow access to non-final local variables from closures? If so: Should it be the default? How do we prevent accidents?

www.javapolis.com

Moving Forward, There are Two Approaches

Gain more experience with prototypes When we come to an informed consensus, start a

narrowly focused JSR This might take a couple of years Precedent established by generics

Start a broadly focused JSR in the near future Must permit any point on the Closure Spectrum First task is to answer the two Big Questions

www.javapolis.com

Closing Sermon

We have a big decision to make! It will have a huge effect on the future of the

Java platform We must take our time and do what is right We must not risk further damage to

“the feel of Java”

Q&AView JavaPolis talks @ www.parleys.com

Thank you for your attention