Next Generation Software Systems and Programming Languages Research
Matthias Felleisen
Our Experience
• PLT has built a bunch of software, including PDE, Web server, IMAP clients, and many more.
• These products share a number of characteristics with other modern software products.
• What does our experience suggest for PL research?
My Thesis, My Goal
• Software systems have become extensible collections of components.
• What does this mean for programming languages: linguistics, pragmatics, virtual machines/compilers, and run-time environments?
Outline
• Two Examples: DrScheme, Web Server
• Language Support for COP• Virtual Machine Support • Contracts for Components
• Summary, Advice
Disclaimer
• PLT’s experience
• My thesis and trajectory
• these are not universal concerns
Examples: Extensible Systems
Two Examples
• DrScheme: a modern, pedagogic programming environment
• The PLT Web Server: an experiment in serving dynamic Web content
DrScheme
• What does it take to build a PDE from scratch?
• How about extensions?
DrScheme scope-sensitive syntax checker
interactive evaluator
algebraic stepper
static debugger
syntax-sensitive editor
Thank You to a Fantastic Team
Cormac, SRC
Shriram Krishnamurthi, Brown
Matthew Flatt, Utah
John Clements, NEU
Robby Findler, UChicago
Paul Steckler, NEU
Philippe Meunier, NEU
Paul Graunke, NEU
DrScheme
Editor
Read-Eval-Print-Loop
model
GUI
view
mutual references
DrScheme: Plugging in Tools
Editor
Read-Eval-Print-LoopTools -- plug-ins
Syntax Checker
Algebraic Stepper
DrScheme: Plugging in Student Programs
Editor
Read-Eval-Print-LoopTools -- plug-ins
Syntax Checker
Algebraic StepperStudent Programs
DrScheme: The Big Picture
Editor
Read-Eval-Print-Loop
a single execution environment: the VM
DrScheme
• powerful plug-ins that interact with editor and read-eval-print loop
• student programs become dynamically plugged in components
• easy communication among all components
A Web Server From Components
• What does it take to build a Web server?
• What about dynamic content?
A Web Server From Components
A server is basically a read-eval-print loop. Upon a TCP connect:
read a request: GET /uniquelylara/paintings/index.html HTTP/1.0 … headers …
interpret the request, the path, and the headers: dispatch on file path, check for defaults, …
print the file: send the requested file back via a TCP write
A Web Server From Components
pattern matching TCP connections URL parsing HTML
Dispatcher
REPL aka Web Server
These components are good old libraries.
A Web Server From Components
pattern matching TCP connections URL parsing HTML
Dis: remote
REPL aka Web Server
Dis: local Dis: dynamic
code duplication
A Web Server From Components
pattern matching TCP connections URL parsing HTML
Dispatch
REPL aka Web Server
Dispatch Dispatchlocal
remote
dynam
these components are parameterized and instantiated
Web Server: Dynamic Content
• dynamic content generation is becoming more and more important
• dynamic content generation is inefficient and slow
• dynamic content generation is difficult to manage
Web Server: Plug-in Scripts
Dispatcher
Server Loop
Scripts
Web Server: Scripts as Components
Dispatcher
Server Loop
dynamically linked
exchange of valuesnot bit streams
Web Server: Scripts as Components
Dispatcher
Server Loop
store script as first-class value
Web Server: Scripts as Components
Dispatcher
Server Loop
link in once
Web Server: Scripts as Components
Dispatcher
Server Loop
link in once, link in twice
Web Server: Scripts as Components
Dispatcher
Server Loop
make the internals of the server available: stack, environment, store
A Web Server From Components
• Our Web server’s performance for dynamic content generation is four times better than that of Apache/Perl and 20 to 40% than mod-Perl.
• It is the only Web server whose scripts are automatically safe for back buttons and cloning: control state.
Other Examples
• Linux:– kernel modules– applications
• Web browsers:– plug-ins– applets
• Multimedia players:– drivers– formats
• MS Office– component: COM
• Java Beans
• .NET
From Component Systems to Programming Languages
From Component Systems to PLs
• component-oriented programming:– different than OOP– different than modules
• VM (virtual machines) for extensible systems of components– more than a virtual machine
• contracts for components
Component Software, Szyperski
• A component is …
– a unit of independent deployment – a unit of third-party composition – a blueprint without state
Szyperski, page 30
COP and Languages: Linguistic Support
• parameterization over– imported components– exchange of values, including critical pieces
• linking– graph-based (mutual references)– hierarchical – dynamic (first-class values in core)
• multi-lingual components
COP and Languages: Virtual Machines
• a VM for many languages
• a VM for dynamic extensions
• a VM for “non-cooperative” extensions
COP and Languages: Quality Assurance
• components exchange values, including objects and functions
• components must protect these values and must promise certain properties– types– (partial) functional specifications– concurrency specifications– QoS guarantees
Linguistic Support for COP
with Matthew Flatt, Robby Findler, Shriram Krishnamurthiand a little bit of help from Dan Friedman
Existing Language Support for Components
Which language constructs should we use to support programming with components?
Existing Language Support for Components
Proposal: Use first-class procedures and arrange them in patterns.
Pro: hierarchical linking, dynamic invoking, by-reference passing work
Cons: the granularity is too far off. Maintaining “linking patterns” by hand is error prone.
Existing Language Support for Components
Proposal: Use classes and objects as components.
Pro: larger granularity, contains many functions, protection, dynamic loading
Cons: The super-class is hard-wired, i.e., no parameterization. It is impossible to wire-in the same class many times. Inheritance poses programming and contract checking problems (Szyperski).
Existing Language Support for Components
Proposal: Use Modula-style modules, Java-style packages, or C#-style assemblies.
Pro: proper granularity
Cons: modules are too static (compile-time). Importing is hard-wired. They are not first-class values.
Existing Language Support for Components
Proposal: Use ML-style functors. They find good uses in SML and OCAML.
Pro: They are parameterized over their importing modules, by definition.
Cons: they are not values. It is not possible to link them in a graphical manner. Like plain modules, they are static and not dynamically linkable.
PLT’s Components: Units
• Units are parameterized over– imported components– references to values
• Units are linked in – graph-based– hierarchical – dynamic manner
Units are first-class values:
Units: A Language Constructs for COP
imports
definitions
exports
Units: A Language Constructs for COP
type t , val g : int -> t
fun f (an-int, a-t) = …
val f : int t -> string
example:
Units: Graphical Linking
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
Units: Hierarchical Linking
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
Units: Dynamic Linking
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = …
f : int t -> string
type t
fun f (an-int, a-t) = … f : int t -> string
Units: Flatt & Felleisen
• PLT’s units support COP well.
• … linking together components is flexible, with rich control
• … widely used in Version 103
• But there are open problems
Units: A Program Design Problem
• Given: a component with – a datatype T with N variants– a collection F of M functions on T
• Wanted: extend T with variants and F with additional functions without access to the source of the component
Krishnamurthi, Felleisen, Friedman
Units: A Program Design Problem
• Why does it matter?– abstract reasons– examples
• How do we do it? – where units work – where we need more
Units Extensibility - Why?
• maintenance: fix once, fix everywhere
• extensibility: add functionality once, add it everywhere
Unit Extensibility: Example 1
• DrScheme: Krishnamurthi et alli the language hierarchy and the set of tools
• datatype is a grammar
• tools: syntax check, type check, stepper, … and more to come
Unit Extensibility: Example 2
• Geometry server: Hudak, Darpa productivity study
• datatype describes arrangements of geometric shapes
• tools: containment, drawing, moving and many others
Unit Extensibility: Example 3
• San Francisco project: Bohrer, IBM
• datatype describes customer, manager, account hierarchies
• tools: back office maintenance tools
Unit Extensibility: The Picture
f : … t … -> … g : … -> … t … h : … t … -> … t …
datatype t
The Core Component
Unit Extensibility: The Picture
f : … t … -> … g : … -> … t … h : … t … -> … t …
datatype t
The Variant Extension
f, g, h : extended
Unit Extensibility: The Picture
f : … t … -> … g : … -> … t … h : … t … -> … t …
datatype t
The Functionality Extension
f, g, h : extended
j : … t … -> … t ..
k : … t … -> … t ..
Unit Extensibility: The Picture
f : … t … -> … g : … -> … t … h : … t … -> … t …
datatype t
The Full Component
f, g, h : extended
j : … t … -> … t ..
k : … t … -> … t ..
self-reference (has-a)
self call
Unit Extensibility: The Experience
• Units work well for extensibility
• … just re-link for each extension
• … each layer around a given component is just an adapter unit (Lieberherr & Lorenz ‘01)
Unit Extensibility: The Experience
• Units alone are not enough
• … need lasses to specify datatypes
• … and mixins for extensions
Unit Extensibility: The Experience
f : … t … -> … g : … -> … t … h : … t … -> … t …
datatype t
The Full Component
f, g, h : extended
j : … t … -> … t ..
k : … t … -> … t ..
Interfaces don’t knowsuperclass -- mixin
Findler & Flatt
Unit Extensibility: The Experience
• Linking with units is complex
• Units do not support multiple languages
• PLT also introduced modules.
Component-Oriented Programming
• modules for multi-lingual programs
• units for complex linking
• mixins for specifying and working with data hierarchies plus implementation inheritance
Components in Languages: Research Problems
• still complex: can we simplify this world?
• practical experience:– units and modules “feel” inside out
• type systems: what will a flexible type system for units and modules look like? – mixins (Krishnamurthi)– sharing by specification (Flatt)
Virtual Machines for COP: Resource Administration
with Matthew Flatt, Robby Findler, Shriram Krishnamurthi
Resource Administration: The Situation
• a dynamic plug-in component is probably third-party code with unknown properties
• several plug-ins may run in parallel
• plug-ins compete for finite resources:– files, database servers– network connections– GUI items
Resource Administration: Example 1
• a Web server responds to many requests simultaneously; each may spawn a Web program:– sharing (communicating) with server– sharing among “scripts”
• a script consumes– time– memory– (references to) server stacks– ports, files, databases, …
Resource Administration: Example 2
• DrScheme runs student programs as dynamic extensions so that tools can directly communicate with executing programs
• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …
Resource Administration: Example 3
• Netscape runs Java applets as dynamic extensions so that …
• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …
Resource Administration: PLs as OSes
• an extensible component system must play OS for its plug-ins
• the underlying VM must provide mechanisms for resource control
• the underlying programming language must “dress” these mechanisms as language constructs
Resource Administration: PLT’s Solution
• dynamic components as threads
• threads are a resource
• all resources are managed in a custodian
Resource Administration: Custodians
current custodian:
thread 1
thread 2
thread 3
network port 1
network port 2
custodianthread 4 file port 3
GUI 1
custodian shutdown
Resource Administration: Experience
• custodians work well in DrScheme and the Web server
• “special” custodians for GUI because we need to manage GUI events (event spaces)
Resource Administration: Problems
• Custodians don’t deal with memory. (Flatt)
• Custodian actions are fixed. Can we specify them?
• Other OS services?
• Update code dynamically? (Krishnamurthi)
Resource Administration
• In the past, advanced languages provided memory administration (garbage collection). Java has introduced GC into the mainstream.
• In the future, advanced languages will provide other resource administration tools. .NET will provide meta-data with assemblies.
Enforcing Invariants in COP
with Robby Findler, Mario Latendresse
Contract for Components: The Problem
The System A Plugin
invariants
Contract for Components: The Problem
The System A Plugin
invariants
The System consists of many components.
If it blows up, who did it? Economics!
Contracts for Components: The Idea
• need behavioral contracts in addition to type signatures for external wires of components
• must monitor these assertions at run-time (or partially verify them)
• assign blame to faulty component
Contracts for Components:Eiffel
• design by contract a la– Eiffel, iContract, jContractor, HandShake,
JVMAssert, …
• get it right, scale to other constructs
• then scale to COP
Contracts for Components: Example
double square_root(double x) { … } @pre: x >= 0 @post: fn result -> result >= 0
double square_root(double x) { … } @pre: x >= 0 @post: fn x -> fn result -> x *x >= result >= 0
partial behavior:
total behavior:
Contracts for Components: Meaning and Blame
• test pre-conditions for method call, blame caller
• test post-condition for method return, blame callee
• deal with invariants for variables similarly
Contracts for Components: Classes and Objects
• classes (or mixins) are in an inheritance hierarchy
• method contracts must be related
• test implications for pre/post conditions at method call & return, blame class extension
Contracts for Components: Functions
• testing predicates on functions is impossible
• use proxy functions that perform tests for first-order types, blame current co-variant party for bad arguments and contra-variant party for bad result
• switch blame positions with applications
Contracts for Components: Experience
• contracts are as important as types
• contracts complement types:– properties of arguments (subset of types)– relating two arguments– relating two functions in a component
Contracts for Components
• OOP and contracts
• FP and contracts
• COP and contracts: – types– linking: graphs, dynamic, hierarchical
Summary and Advice
Software Systems
• build large systems from components– in-house– third party– different sizes
• plug-in components statically
• plug-in components at run-time
Software Systems
• systems have complex interactions
• systems act as quasi-OS’es for dynamic plug-ins
• such systems consist of components in many languages
Programming Languages
• provide VMs that support high-level values and value sharing
• provide VMs that support resource administration for non-collectible resources
• provide VMs that accommodate many languages
Programming Languages
• provide constructs for constructing and linking components of all sizes and all shapes
• provide constructs for dynamic extensions, support mini-OSes for COP
Programming Languages
• provide interfaces with contracts
• check contracts in a well-founded manner
• assign blame properly
PLT’s Solutions
• modules, units, & mixins are a good solution, but not the last word
• custodians are a good start toward resource administration, but not the last word
• and this slide isn’t the last word
Advice for Academics: PL Research
• build systems, not (just) compilers:• useful and team-sized, e.g., DrScheme • useful and pair-sized, e.g., Web server• useful individual scripts• get “customers”
• test academic languages with all kinds of system; use feedback for PL design
Advice for Industry
• the JVM was a start; it does not accommodate other languages easily, it does not administrate resources
• the .NET world is made for other languages; it still lacks resource administration
• these machines are not the last word, they are just a start for COP
now go forth and do some more type research
Top Related