3 years with Clojure
-
Upload
michael-klishin -
Category
Software
-
view
2.027 -
download
1
description
Transcript of 3 years with Clojure
Life of an immutant3 years with Clojure
About me
• @michaelklishin
• github.com/michaelklishin
Setting expectations
Setting expectations
• This is an experience report
Setting expectations
• This is an experience report
• Not a language primer
Setting expectations
• This is an experience report
• Not a language primer
• Assumes no familiarity except basic facts
Setting expectations
• This is an experience report
• Not a language primer
• Assumes no familiarity except basic facts
• ~200 slides. This is gonna be fast.
Background
Background
• Data processing
Background
• Data processing
• Scheduling systems
Background
• Data processing
• Scheduling systems
• Specialized Web crawlers
Background
• Data processing
• Scheduling systems
• Specialized Web crawlers
• Billing, accounting
Background
• Data processing
• Scheduling systems
• Specialized Web crawlers
• Billing, accounting
• Infrastructure automation
Wanted
Wanted
• An environment that does not suck
Wanted
• An environment that does not suck is operationally sane
Wanted
• An environment that does not suck is operationally sane
• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)
Wanted
• An environment that does not suck is operationally sane
• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)
• A runtime that yields good baseline performance
Wanted
• An environment that does not suck is operationally sane
• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)
• A runtime that yields good baseline performance
• Painless upgrade paths
Wanted
• Community
Wanted
• Community
• Benefit from existing libraries and tools
Wanted
• Community
• Benefit from existing libraries and tools
• Concurrency and parallelism
Wanted
• Community
• Benefit from existing libraries and tools
• Concurrency and parallelism
• A productive, concise language
Wanted
JVM
JVM
JVM
• One of the most mature runtimes you can find
JVM
• One of the most mature runtimes you can find
• Excellent tooling
JVM
• One of the most mature runtimes you can find
• Excellent tooling (Visual VM, YourKit, jstack, jhat, VM/+XX flags, BTrace, …)
JVM
• One of the most mature runtimes you can find
• Excellent tooling
• Excellent baseline performance
JVM
• One of the most mature runtimes you can find
• Excellent tooling
• Excellent baseline performance
• Multilingual: Clojure, Java, JRuby, Scala, JS implementations
Clojure
Clojure
• Very high level, concise, expressive
Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed
Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed
• Decent tooling
Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed
• Decent tooling
• Fun to work with (motivation)
Clojure
• No bored academics, built by a practicioner
Clojure: concurrency/parallelism story
Clojure: concurrency/parallelism story
• Immutable data structures by default
Clojure: concurrency/parallelism story
• Immutable data structures by default
• Identity/values separation
Clojure: concurrency/parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types
Clojure: concurrency/parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types
• java.util.concurrent
Clojure: concurrency/parallelism story
When I find myself in times of trouble Prof. Doug Lea comes to me Coding lines of wisdom j.u.c.
Clojure: concurrency/parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types
• java.util.concurrent
• Reactor, Akka, Disruptor, RxJava…
Clojure: concurrency/parallelism story
• Async I/O: NIO, Netty, Vert.x…
Clojure: concurrency/parallelism story
• You have choice
Clojure: concurrency/parallelism story
• You have choice
• Runtime parallelism (on JVM, CLR)
Clojure: concurrency/parallelism story
• You have choice
• Runtime parallelism (on JVM, CLR)
• Hundreds, thousands of threads at the same time
Clojure: baseline performance
Clojure: baseline performance
• What matters the most is what bytecode the compiler produces
contextneeded.com
Clojure: baseline performance
• What matters the most is what bytecode the compiler produces
• Java interop bytecode is typically the same as produced by javac
Clojure: baseline performance
• What matters the most is what bytecode the compiler produces
• Java interop bytecode is typically the same as produced by javac
• Watch out for reflective calls, primitive boxing
Clojure: baseline performance
• Optional type hints
Clojure: baseline performance
• Optional type hints
• Type inference in simple cases
Clojure: baseline performance
(let [d (java.util.Date.)] d)
Clojure: baseline performance
(let [d (java.util.Date.)] d)
Clojure: baseline performance
• Optional type hints
• Type inference in simple cases
Clojure: baseline performance
• Optional type hints
• Type inference in simple cases
• If JVM can optimize your bytecode, things run screaming fast
Clojure: baseline performance
• Optional type hints
• Type inference in simple cases
• If JVM can optimize your bytecode, things run screaming fast
• From 2% (Java lib wrappers) to 200% (naïve numerics heavy code, tight loops) penalty compared to Java
Clojure: baseline performance
• Immutable data structures have O(log32 n) access complexity
Clojure: baseline performance
• Immutable data structures have O(log32 n) access complexity
• Immutable data structures have associated GC tax
Clojure: baseline performance
• Immutable data structures have O(log32 n) access complexity
• Immutable data structures have associated GC tax
• Focus on exploiting parallelism over squeezing µs from a single thread
Clojure: baseline performance
• Poor startup time, excellent execution performance
Clojure: baseline performance
• Poor startup time, excellent execution performance
• Anecdotal evidence: 10 — 50 times throughput improvements compared to Ruby in real apps
Clojure: stability
Clojure: stability
• Have been using 1.3 since alpha1
Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years
Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth fixing (I did something really stupid)
Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth fixing (I did something really stupid)
• Alphas more solid than GA releases of almost all other languages I’ve worked with
Clojure: stability
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth fixing (I did something really stupid)
• Alphas more solid than GA releases of almost all other languages I’ve worked with
• Others (e.g. ThoughtWorks) report the same experience
Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.
Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.
• When upgrading is this easy, people do it happily
Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.
• When upgrading is this easy, people do it happily
• No library binary compatibility hell some other JVM languages suffer from
Clojure: stability
• Many active developers are conservative with versioning & calling things “done”
Clojure: stability
• Many active developers are conservative with versioning & calling things “done”
• Fits JVM ecosystem maturity really well
Clojure: stability
• The real reason: Clojure is small and was designed to be simple
Clojure: stability
• No monkey patching (there are features that give you all the good parts of teh)
Clojure: consistency
Clojure: consistency
• Collections/sequences library is very unified
Clojure: consistency
• Collections/sequences library is very unified
• Easy to implement your own data types that work just like core Clojure types
Clojure: consistency
• Collections/sequences library is very unified
• Easy to implement your own data types that work just like core Clojure types
• Almost no “bad baggage” in the core
Clojure: consistency
• Collections/sequences library is very unified
• Easy to implement your own data types that work just like core Clojure types
• Almost no “bad baggage” in the core
• The language was actually designed
Clojure: deployment
Clojure: deployment
• Pretty standard JVM deployment options
Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc
Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc
• A tiny shell script + start-stop-daemon
Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc
• A tiny shell script + start-stop-daemon
• Chef or Capistrano
Clojure: deployment
• CloudFoundry
Clojure: deployment
• CloudFoundry
• Heroku
Clojure: deployment
• CloudFoundry
• Heroku
• Immutant/JBoss AS
Clojure: deployment
• CloudFoundry
• Heroku
• Immutant/JBoss AS
• OpenShift
Clojure: operations
Clojure: operations
• VisualVM, jstack, jheap, …
Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL
Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL
• Remote REPL over HTTP(S)
Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL
• Remote REPL over HTTP(S)
• Not quite Erlang but still impressive
Clojure: operations
• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate
Clojure: operations
• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate
• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)
Clojure: operations
• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate
• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)
• BTrace
Clojure: community
Clojure: community
• Full of intelligent engineers who don’t give two shits about Hacker News
Clojure: community
• Full of intelligent engineers who don’t give two shits about Hacker News
• Over 10 conferences to date
Clojure: community
• Full of intelligent engineers who don’t give two shits about Hacker News
• Over 10 conferences to date
• ~10K people on the mailing list
Clojure: community
• Full of intelligent engineers who don’t give two shits about Hacker News
• Over 10 conferences to date
• ~10K people on the mailing list
• Users from small and not-so-small startups to Fortune 50 corporations in finance, biotech, retail
Clojure: fun to work with
Clojure: fun to work with
• Profoundly changes the way you think
Clojure: changes the way you think
Clojure: changes the way you think
• About managing your program state
Clojure: changes the way you think
• About managing your program state
• About the value of experimenting (REPL driven development)
Clojure: changes the way you think
• About managing your program state
• About the value of experimenting (REPL driven development)
• About metaprogramming
Clojure: changes the way you think
• About managing your program state
• About the value of experimenting (REPL driven development)
• About metaprogramming
• About isolating side effects
Clojure: changes the way you think
Clojure: changes the way you think
• Program = pure core + parts that communicate with the outside world (network, disk I/O)
Clojure: changes the way you think
• Program = pure core + parts that communicate with the outside world (network, disk I/O)
• Pure parts are ridiculously easy to reason about, test, experiment with and reuse
Clojure: changes the way you think
• Program = pure core + parts that communicate with the outside world (network, disk I/O)
• Pure parts are ridiculously easy to reason about, test, experiment with and reuse
• Typical Rails app: 95% of methods have side effects
Clojure: changes the way you think
• Typical Clojure app: 40% of functions have side effects
Clojure: changes the way you think
• Typical Clojure app: 40% of functions have side effects
• Anecdotal evidence: 3-4 times less time spent on tests
Clojure: changes the way you think
• Typical Clojure app: 40% of functions have side effects
• Anecdotal evidence: 3-4 times less time spent on tests
• Moral of the story: make more of your code not have side effects
Clojure: fun to work with
• Profoundly changes the way you think
Clojure: fun to work with
• Profoundly changes the way you think
• Motivates you to learn
Clojure: fun to work with
• Profoundly changes the way you think
• Motivates you to learn
• Easy things are easy, hard things are possible
Clojure: fun to work with
• Profoundly changes the way you think
• Motivates you to learn
• Easy things are easy, hard things are possible
• Wheels are already invented in clojure.core or JDK
Clojure: fun to work with
• Stable: fewer distractions, get in the flow
Clojure: fun to work with
• Stable: fewer distractions, get in the flow
• REPL driven development keeps you in the flow for longer periods
Clojure: fun to work with
• Stable: fewer distractions, get in the flow
• REPL driven development keeps you in the flow for longer periods
• Polymorphism done right <3
Clojure: fun to work with
“Clojure demands that you raise your game and rewards you greatly for it…”
Clojure: fun to work with
• The community is full of smart and down to earth people.
Clojure: use cases
Clojure: use cases
• Data processing
Clojure: use cases
• Data processing
• Stream processing
Clojure: use cases
• Data processing
• Stream processing
• Micro-services
Clojure: use cases
• Data processing
• Stream processing
• Micro-services
• Machine learning
Clojure: use cases
• Data processing
• Stream processing
• Micro-services
• Machine learning
• Web development
Clojure: misc
Clojure: misc
• Leiningen (leiningen.org) is excellent
Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks are just Clojure functions
Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks are just Clojure functions
• Packages your code + all deps into a single jar for deployment
Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks are just Clojure functions
• Packages your code + all deps into a single jar for deployment
• Makes it trivial to develop against multiple Clojure versions
Clojure: misc
• clojars.org
Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS
Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS
• Very easy to deploy snapshot releases
Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS
• Very easy to deploy snapshot releases
• Clojars and Leiningen are maintained by the same core team
Clojure: misc
• travis-ci.org support
Clojure: misc
• travis-ci.org support
• Test against multiple JDKs
Clojure: ecosystem
• Feels a bit like Ruby circa 2009
Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your reach
Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your reach
• And Leiningen + clojars.org
Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your reach
• And Leiningen + clojars.org
• Community ~ doubles every year in size
Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your reach
• And Leiningen + clojars.org
• Community ~ doubles every year in size
• Google for “The State of Clojure” survey
Clojure: ecosystem
• ~35% of people come from Java, ~35% from Ruby or Python
Clojure: ecosystem
• ~35% of people come from Java, ~35% from Ruby or Python
• Libraries, not frameworks
Clojure: ecosystem
• ~35% of people come from Java, ~35% from Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every problem
Clojure: ecosystem
• ~35% of people come from Java, ~35% from Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every problem
• 15 books, 2 outdated (cover 1.2)
Clojure: ecosystem
• ~35% of people come from Java, ~35% from Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every problem
• 15 books, 2 outdated (cover 1.2)
• clojure-doc.org, tryclj.com, …
Clojure: ecosystem
Clojure: what sucks
Clojure: what sucks
• clojure.org
Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly OK, books are great)
Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly OK, books are great)
• Compiler messages can be cryptic
Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly OK, books are great)
• Compiler messages can be cryptic
• JVM startup time + code compilation on each run (without REPL or tools such as Drip)
Clojure: what sucks
• Paper contributor agreement
ClojureWerkz
ClojureWerkz
“A growing collection of open source Clojure libraries…”
ClojureWerkz
“It just werkz…”
ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented
ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented
• Friendly license (EPL, the same as Clojure)
ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented
• Friendly license (EPL, the same as Clojure)
• Tested against multiple Clojure versions, 3 JDKs on travis-ci.org
ClojureWerkz
clojurewerkz.org
ClojureWerkz
• Projects reuse a lot of Java libraries
ClojureWerkz
• Projects reuse a lot of Java libraries
• Which are often officially supported (e.g. RabbitMQ, ElasticSearch, MongoDB, Riak, Reactor)
“ClojureWerkz stuff is really improving my Clojure experience, which is rapidly becoming my language of choice…”
Clojure: the takeaway
Clojure: the takeaway
• A lot of merit of its own
Clojure: the takeaway
• A lot of merit of its own
• Great stability
Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing
Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing
• Don’t fear the JVM
Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing
• Don’t fear the JVM
• Immutability is essential for sane concurrent programming
Clojure: the takeaway
• Clojure greatly rewards those who choose to use it
The real reason?
The real reason?
???
clojure.core/lazy-cat
Thank you
• @michaelklishin
• github.com/michaelklishin
• clojurewerkz.org