Fpx talk 2014

NameChris Marshall @oxbow_lakes

GSA Capital Partners LLPMarch 2014

Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority.

Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261

Wrapping an imperative API in a functional one

Mathematics Degree

Working in financial software since 1999

• Smalltalk for ~6 months

• Java thereafter

• Scala since Dec 2008

JP Morgan for 6 years

• ~200,000 employees

GSA Capital for 7 ¾ years

• Quant hedge fund

• ~130 employees

Backgroundwho am i

Private & Confidential. Not for distribution. GSA Capital Partners LLP is authorised and regulated by the Financial Services Authority. Registered in England & Wales. Stratton House, 5 Stratton Street, London, W1J 8LA. Number OC309261

Low-latency links & feeds

• Up to 800,000 trades /day

• 108 market events / day

Historic / Current Market Data

• Listing changes (e.g. SUNW.O becomes JAVA.O becomes ORCL.O)

• News Events

Backtesting / Execution Framework

Everything Else

• This is where I come in

• What are our positions? What is our P&L?

• Are our trades reconciled?

• Reporting (brokers, regulators, administrators)

GSAWhat do we do? Roughly half the company are technologists

Embarrassingly basic stuff• Target audience (people who’ve not done this before)• Scala programmers working on the JVM

Scala• I’m going to assume you can read scala code• I’m going to assume familiarity with typeclasses and their formulation in scalaz• I’m going to assume you understand implicits

Overview• Outline of (simplified) imperative API• Wrapping in functional scala• A tangent on futures• An example• Extending the example

This talk

Furnace• Furnace is one of GSA’s central systems• It’s a trade capture system which contains all of GSA’s daily trading activity• Currently handles flow of up to 800k trades/day• Plus many more cash events • Including state-transitions (modifications, confirmations etc), it is growing by ~2m events per day• Here’s an extremely simplified view of what its (Java) client API looks like

Furnace: an imperative Java data API

interface FurnaceService { FurnaceSession createSession(String, String); //user, password }

interface FurnaceSession { Iterator<FurnaceEvent> replayEvents(Date); Future<Long> insertEvent(FurnaceEventBuilder); void close(); }

Java/Scala interop• Scala’s Java interop means this is pointless, right?

Rationale• furnace’s API provides a very low-level view of interacting with the system• ARM• Idiomatic property access• For the hell of it

Typical Query• Give me a Map[TxnId, Event] where the values are the latest event on each transaction, for those

transactions whose latest event is not in state DELETED

Why implement a wrapper at all?

The Basic Wrapper

A datatype “Furnace”• Representing the calculation of a value from the interaction• Obviously, this should be a functor and a monad• We should be able to put an arbitrary value inside our “program”• We need to be able to derive a value from a session

sealed trait Furnace[+A] { def unsafePerformIO(): A

def map[B](f: A => B): Furnace[B] def flatMap[B](f: A => Furnace[B]): Furnace[B] }

object Furnace { def point[A](a: => A): Furnace[A] def apply[A](f: FurnaceSession => A): Furnace[A] }

StackOverflowException

Trampolines• Go and look at how scalaz 7 defines IO• Ivory Towers!

import scalaz._; import Scalaz._; import Free._

sealed trait Furnace[+A] { ... private def apply(rw: FurnaceSession): Trampoline[(FurnaceSession, A)] }

(I don’t follow this either)

Now define some more stuff• A trampolining IO operation• Implement the module “constructors”

trait FurnaceFunctions { def furnace[A](f: FurnaceSession => Trampoline[(FurnaceSession, A)]) = new Furnace[A] { private def apply(rw: FurnaceSession) = Suspend(() => f(rw)) } }

object Furnace extends FurnaceFunctions { def point[A](a: => A) = apply(_ => a) def apply[A](f: FurnaceSession => A) = furnace(rw => return_(rw -> f(rw))) }

“rw” is Real World

Now for map/flatMap• They make use of the previously defined furnace and apply

sealed trait Furnace[+A] { ...

def map[B](f: A => B) = furnace(rw => apply(rw) map { case (nw, a) => (nw, f(a)) })

def flatMap[B](f: A => Furnace[B]) = furnace(rw => apply(rw) flatMap { case (nw, a) => f(a)(nw) }) }

Getting a value out

unsafePerformIO• Resource management

object Furnace { ... def unsafePerformIO[A](fa: Furnace[A])(implicit l: Login): A = { val s = new FurnaceService //simplifying val t = s.createSession(l.user, l.password) try fa.apply(t).run._2 finally t.close() } }

Typeclasses

object Furnace { module => ... implicit val FurnaceInstances = new Monad[Furnace] { def point[A](a: => A) = module.point(a) def bind[A, B](fa: Furnace[A])(f: A => Furnace[B]) = fa flatMap f } }

That was it?

We can now start defining some useful methods• Querying

package gsa.furnace

package object dsl { def dateQuery(date: Date): FurnaceSession => List[FurnaceEvent] = fs => { import collection.JavaConverters._ fs.replayEvents(date).asScala.toList } }

Useful wrappers

Decorate the basics

implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => list.foldLeft(Map.empty[Long, FurnaceEvent]) { (m, ev) => m + (ev.getTransactionId -> ev) } filter { case (_, v) => excludeDeleted implies (ev.getState =/=

DELETED) } }

Where have we got to?

A simple program• Report is cheating a bit – probably should interleave with the IO monad• The expression inside the unsafePerformIO is a Furnace[Unit]

def report(evs: Iterable[FurnaceEvent]): Unit = //create a file, send an email?

import gsa.furnace.dsl._ val x = Furnace(dateQuery(today)).latest().map(xs => report(xs.values)

implicit val login = Login(“cmarsha”, “CorrectBatteryHorseStaple”) Furnace.unsafePerformIO(x)

Updating

More useful methods• Inserting

package gsa.furnace

package object dsl { ...

def insert(ev: FurnaceEventBuilder): FurnaceSession => j.u.c.Future[j.l.Long] = fs => fs.insert(ev) }

Still vestiges of Java

We are still working with j.u.c.Future• It’s actually rather trivial to create a “listenable” future• If you control the mechanism of submission (Use CompletionService and inform the listeners on the

completion thread)

We want to use s.c.Future• Must implement a few methods, isCompleted, onComplete, ready, result, value• Easy to convert a Listenable future into a s.c.Future

def onComplete[U](f: Try[T] => U)(implicit e: ExecutionContext) = { val l = new FutureListener[T] { def onCancelled() = try f(Failure(new CancellationException)) finally

removeListener(this) def onResult(r: T) = try f(Success(r)) finally removeListener(this)

def onThrowable(t: Throwable) = try f(Failure(t)) finally removeListener(this) } addListener(l) }

Our victory is complete!

We modify our wrapper• We first persuade the Furnace library author to give us this!

• Then we modify our API to remove the last traces of the old order

package gsa.furnace package object dsl { ...

def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long] = fs => fs.insert(ev).asScala.map(_.longValue) }

interface FurnaceSession { ... ListenableFuture<j.l.Long> insertEvent(FurnaceEventBuilder); }

Aside: Scalaz s.c.Future instances

implicit def FutureInstances(implicit EC: ExecutionContext) = new Monad[Future] with Traverse[Future] { def point[A](a: => A) = Promise.successful(a).future

override def map[A, B](fa: Future[A])(f: A => B) = fa map f

override def ap[A, B](fa: => Future[A])(fab: => Future[A => B]) = (fa zip fab) map { case (a, f) => f(a) } //for (f <- fab; a <- fa) yield f(a)

def bind[A, B](fa: Future[A])(f: A => Future[B]) = fa flatMap f

def traverseImpl[G[_], A, B](fa: Future[A])(f: A => G[B]) (implicit A: Applicative[G]): G[Future[B]] = A.map(f(Await.result(fa, Duration.Inf)))(b => point(b))

Where were we again?• We’ve seen how we could query furnace in a more “functional” style• We were removing Java futures, which were returned by the insert method and replacing them with scala

futures

Why?• Well, we can now compose a “pipeline” which involves querying and updating (inserting) events• What does one of those look like?

Quick recap

A “pipeline” where we query/update

A look at the finished article

type CorrectedEvents = s.c.Future[Stream[Long]]val correctFxRates: Map[Currency, Double] = ???def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Set[Currency]

val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z

Await.result(Furnace.unsafePerformIO(pipeline)(login), atMost = 5.minutes)

Let’s home in on correct_

type CorrectedEvents = s.c.Future[Stream[Long]]

//recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long]

def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents]

= xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[Furnace[Future[Long]]]

= xs.toStream.map(e => Furnace(insert(e.withFx(rates(e.ccy)))).sequenceU //<error>: is a Furnace[Stream[Future[Long]]]

= xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy)))) //<error>: is a Furnace[Stream[Future[Long]]]

= _.map(_.sequenceU) //is a Furnace[Future[Stream[long]]] is a Furnace[CorrectedEvents]

But we can pretty this last bit up

Another implicit wrapper later

• We can now implement correct_ as follows

implicit class FurnaceFutureStreamW[A](f: Furnace[Stream[Future[A]]]) { def sequenced(implicit EC: ExecutionContext) = f map (Future.sequence(_))}

def correct_(xs: Iterable[FurnaceEvent] ], rates: Currency => Double) = xs.toStream.traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))).sequenced

A “pipeline” where we query/update

val pipeline: Furnace[CorrectedEvents] = for { x <- Furnace(dateQuery(d)).latest() //x is Map[Long, FurnaceEvent] y <- suspiciousFxRates(x.values).point[Furnace] z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace] } yield z

Error Handling

Recall the type of our pipeline

• We “cheated” by using inserting an exception into our future• What if dealing with furnace throws exceptions?• What if computing our suspicious FX rates throws exceptions?

val pipeline: Furnace[CorrectedEvents]

def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Exception \/ Set[Currency]

Monad Transformers: EitherT

What’s one of those then?• Monads do not compose (an M[N[A]] is not a monad, even if both M and N are)

If Banana is a monad with the correct structure• We denote its transformer as BananaT• Then for any monad M, M[Banana[A]] gives us a BananaT[M, A]• BananaT is a monad• BananaT looks like a Banana

There are monad transformers for many monads in Scalaz• StreamT, ListT, OptionT, StateT, ReaderT, EitherT• But not all monads (no FutureT, IOT)

Error Handling

Adding error handling to the furnace wrapper• Pretty trivial

object Furnace { ... def eitherT[A](fa: FurnaceSession => A): EitherT[Furnace, Throwable, A] = EitherT(apply(fs => \/.fromTryCatch(fa(fs)))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Furnace[Throwable \/ A] }

A “pipeline” handling errors

type CorrectedEvents = s.c.Future[Stream[Long]]def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable \/ Set[Currency]

val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

Furnace.unsafePerformIO(pipeline.run)(login) // Exception \/ CorrectedEvents

Let’s home in on correct_ (again)

type CorrectedEvents = s.c.Future[Stream[Long]]

//recall wrapper package object gives us def insert(ev: FurnaceEventBuilder): FurnaceSession => s.c.Future[Long]

def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents]

= xs.toStream.map(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a Stream[EitherT[Furnace, Throwable, Future[Long]]]

= xs.toStream.traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) //<error>: is a EitherT[Furnace, Throwable, Stream[Future[Long]]] // Furnace[Throwable \/ Stream[Future[Long]]]

= _.map(_.sequenceU) //is a EitherT[Furnace, Throwable, Future[Stream[long]]] //is a EitherT[Furnace, Throwable, CorrectedEvents]

What kind of wizardry is this?

//Before def correct_(xs: Iterable[FurnaceEvent], rates: Currency => Double): Furnace[CorrectedEvents] = xs.toStream .traverseU(e => Furnace(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU) //After def correct_(xs: Iterable[FurnaceEvent] , rates: Currency => Double): EitherT[Furnace, Throwable, CorrectedEvents] = xs.toStream .traverseU(e => Furnace.eitherT(insert(e.withFx(rates(e.ccy))))) .map(_.sequenceU)

A “pipeline” handling errors

type CorrectedEvents = s.c.Future[Stream[Long]]def suspiciousFxRates(xs: Iterable[FurnaceEvent]): Throwable \/ Set[Currency]

val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { w <- Furnace.eitherT(dateQuery(d)) x <- EitherT.right(w.point[Furnace].latest()) y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- if (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

Furnace.unsafePerformIO(pipeline.run)(login) // Exception \/ CorrectedEvents

What I’ve shown you• We’ve scala-fied a Java API• We’ve removed resource management from a programmers’ concern• We can create blocks of reusable, composable code• We have been able to take advantage of the higher-order abstractions provided by libraries like scalaz

What I’ve not shown you• The library throttles the rate at which updates can be made, all transparently to the client• Using other monad transformers with the library (like StreamT)• Using the library with Monoids• Lots more wrappers• Value classes (unboxed typesafe primitives) for transaction id, event id, String properties etc

What you might do yourself• Implement ListenableFutureExecutorService which returns listenable futures• Given a listenable future, implement map and flatMap. It’s not as trivial as it might first appear.

So what was the point?

Extra – abstracting over “latest”

We couldn’t call latest on EitherT• Or OptionT, StreamT etc.• We’d need to declare multiple implicit conversions for each type, wouldn’t we?

• Wouldn’t we?

implicit class FurnaceW(f: Furnace[List[FurnaceEvent]]) { def latest(excludeDeleted: Boolean = true): Furnace[Map[Long, FurnaceEvent]] = f map { list => ... } }

We wouldn’t

We can abstract over a functor of List of events• I’ll see your type-fu and raise you• Thanks, Kenji Yoshida

final class ListOfEventFunctorOpsW[F[_]: A](self: F[A], F: Functor[F]) { def latestF(excludeDeleted: Boolean = true) (implicit A: A =:= List[FurnaceEvent]): F[Map[Long, FurnaceEvent]] = F.map(self) { a => A(a) ... } }

implicit def listOfEventFunctorOps[FA](fa: FA)(implicit F: Unapply[Functor, FA])

= new ListOfEventFunctorOps[F.M, F.A](F(fa), F.TC) ^^^ ^^^ dependent types

And now, eternity

val pipeline: EitherT[Furnace, Throwable, CorrectedEvents] = for { x <- Furnace.eitherT(dateQuery(d)).latestF() y <- EitherT(suspiciousFxRates(x.values).point[Furnace]) z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else EitherT.left((new MissingFxException).point[Furnace]) } yield z

x <- Furnace(dateQuery(d)).latest() y <- suspiciousFxRates(x.values).point[Furnace] z <- (y forall correctFxRates.keySet) correct_(x.values filter (y contains _.ccy), correctFxRates) else Promise.failed(new MissingFxException).future.point[Furnace]

StackOverflows - approaches

final class SequencedFunctorOps[F[_], A](self: F[A], F: Functor[F]) { def sequencedF[B, M[_] <: TraversableOnce[_]] (implicit EC: ExecutionContext, A: A => M[Future[B]], cbf: CanBuildFrom[M[Future[B]], B, M[B]]): F[Future[M[B]]] = F.map(self)(a => Future.sequence(A(a))) }

object fewchaz { object Implicits { object sequential { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* do not override ap */ } } object parallel { implicit def FutureApplicative(implicit EC: ExecutionContext) = { /* override ap */ } } } } import fewchaz.Implicits.sequential._

ReaderWriterStateT

type R = SmartProps type W = IO[Unit] //or any monoid type S = Unit type F[+A] = Furnace[A] type E = ErrorMsg type M[+A] = ReaderWriterStateT[F, R, W, S, A]

type Program[+A]= EitherT[M, E, A]

object Program { def furnace[A](f: R => Furnace[E \/ A]): Program[A] = EitherT[M, E, A](ReaderWriterStateT[F, R, W, S, E \/ A] { case (r, s) => f(r).map(or => (IO(()), or, s)) } )

def apply[A](f: R => E \/ A): Program[A] = furnace(f andThen (_.point[F]))

def either[A](value: E \/ A): Program[A] = apply(_ => value)

def fromTryCatch[A](value: => A): Program[A] = either(try \/-(value) catch { case NonFatal(e) => -\/(ErrorMsg(e)) })

def report(msg: String): Program[Unit] = EitherT.right[M, E, Unit](ReaderWriterStateT[F, R, W, S, Unit] { case (r, s) => (IO(println(msg)), (), s) }) }

...ReaderWriterStateT

lazy val program = for { _ <- fromTryCatch(ProductUniverse.setUniverseProvider(new CoreUniverseProvider)) _ <- report(s"About to run the program with: [${args mkString " "}]") dd <- date _ <- report(s"Running for carry date [$dd]") ps <- positions(dd) _ <- report(f"Found ${ps.size}%,d positions in DI1 futures") cc <- //Only try and get marks and D1 if there *are* positions ps.nonEmpty ?? (for { ms <- marks(dd) _ <- report(f"Found ${ms.size}%,d marks on DI1 futures") d1 <- d1Rate(dd) _ <- report(s"D1 rate is $d1") xx <- either(costs(dd, ps.values.toStream, ms)(d1)) } yield xx) _ <- report(s"Costs for positions are: $cc") fs <- //Must run the furnace section regardless of whether we now have costs furnaceInsertOrUpdate(cc, dd) rs <- { import scala.concurrent.duration._ fromTryCatch(Await.result(fs, atMost = 5.minutes)) } _ <- report(f"Created ${rs.size} furnace events") } yield ()

Nothing but a Gnab Gib

case class Result(log: W, v: E \/ Unit)

val result = (for { c <- config //E \/ SmartProps l <- c.maybeLogin(prefix = Some("furnace")).missingConfig //E \/ Login r <- \/-(Furnace.unsafePerformIO(program.run.run(c, ()))(l)) } yield Result(r._1, r._2)) valueOr { s => Result(IO(println(“Unable to invoke program”)), -\/(s)) }

//The end of the world (result.log |+| IO(println(result.v))).unsafePerformIO

This presentation

• http://www.slideshare.net/oxbow_lakes/fpx-talk-2014

• Fewchas: https://gist.github.com/oxbowlakes/8666295

• Furnace Scala API: https://gist.github.com/oxbowlakes/8686567

References

• Runar Bjarnason talk: http://skillsmatter.com/podcast/scala/stackless-scala-free-monads

• Scalaz: https://github.com/scalaz/scalaz

Extrasfor the masochists

Contact us

GSA Capital Partners LLP

investor.relations@gsacapital.comT +44 (0)20 7959 8850

London Office

Stratton House5 Stratton StreetLondon W1J 8LA

T +44 (0)20 7959 8800F +44 (0)20 7959 8845

New York Office

1140 Ave of the Americas9th FloorNew York NY 10036

Fpx talk 2014

Economy & Finance

Transcript of Fpx talk 2014

FPX NICKEL CORP. (FORMERLY FIRST POINT MINERALS CORP ... · FPX NICKEL CORP. (FORMERLY FIRST POINT MINERALS CORP.) MANAGEMENT’S DISCUSSION & ANALYSIS FORM 51-102F1 for the three

FPX Series Fiber Panels - CommScope.com · 6 FPX Series Fiber Panels MPO Cassette Optical Specifications and Polarity FPX cassettes are wired straight through as per the recommendations

Metalux FPX Surface kit installation instructions · Instrucciones de instalación – Kit de superficie de panel FPX Risk of Fire, Electrical Shock, Cuts or other Casualty Hazards-

Fpx Poster English Low

Fpx Catalog

FPX Nickel Corp. – An Undervalued Pure Nickel Play · 2019. 3. 26. · 3/25/2019 FPX Nickel Corp. – An Undervalued Pure Nickel Play - Junior Stock Review ...

Remote Management of the Field Programmable Port Extender (FPX)

FPX Clusterstory 7

FPX Clusterstory 1

FPX THOUGHT LEADERSHIP Rule-Based Conﬁguration Configuration_White Paper_Final.pdfRule-Based Conﬁguration FPX THOUGHT LEADERSHIP FPX.COM | (866) 826-6344 | 8000 NORMAN CENTER DR,

Field Programmable Port Extender (FPX) 1 NCHARGE: Remote Management of the Field Programmable Port Extender (FPX) Todd Sproull Washington University, Applied.

FPX Clusterstory 5

FORCE TENTM FDX/FPX OPERATION MANUAL€¦ · FDX OPERATION MANUAL TABLE OF CONTENTS Section Page Precautions 1 Table of Contents 2 FDX / FPX Features 3 FPX Application 6 FDX / FPX

Panasonic Fpx Catalogue

FORCE TENTM FDX/FPX OPERATION MANUAL · 2020. 3. 18. · FORCE FPX FORCE TENTM FDX/FPX OPERATION MANUAL CALL TOLL FREE: 800 345 4188 sales@wagnerinstruments.com Wagner Instruments

2014 pycon-talk

User manual – UM Student E-Payment Portal (FPX)

FPX Clusterstory 9

Control Processor Switch Fabric ATM Switch Core Port Processors FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP FPX SPC LC IPPOPP.

FPX Series Fiber Panels - CommScope · 2 FPX Series Fiber Panels CommScope’s FPX series provides industry-leading fiber cable protection and management in a modular footprint. With