Designing a reactive data platform: Challenges, patterns, and anti-patterns
-
Upload
alex-silva -
Category
Software
-
view
399 -
download
0
Transcript of Designing a reactive data platform: Challenges, patterns, and anti-patterns
![Page 1: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/1.jpg)
DESIGNING A REACTIVE DATA PLATFORM:
CHALLENGES, PATTERNS AND
ANTI-PATTERNS
Alex Silva
![Page 2: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/2.jpg)
![Page 3: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/3.jpg)
![Page 4: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/4.jpg)
Me!
Me!Me!
![Page 5: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/5.jpg)
![Page 6: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/6.jpg)
![Page 7: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/7.jpg)
![Page 8: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/8.jpg)
![Page 9: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/9.jpg)
Distributed
Elastic
LocationAgnostic
Open
MessageDriven
Self-Healing
![Page 10: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/10.jpg)
REACTIVE
![Page 11: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/11.jpg)
The Reactive Manifesto
Responsive Elastic
ResilientMessageDriven
![Page 12: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/12.jpg)
Responsiveness
![Page 13: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/13.jpg)
Elasticity
![Page 14: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/14.jpg)
Scaling
OUTScaling
UPVS
![Page 15: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/15.jpg)
Elasticity
Asynchronous ShareNothing
Divide andConquer
LocationTransparency
![Page 16: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/16.jpg)
Synchronous Messaging
Inherit ordering introducesimplicit back pressure on the sender
3
1
2
Synchronous
![Page 17: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/17.jpg)
4Invalid!
Asynchronous
1
2
3
Asynchronous Messaging
![Page 18: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/18.jpg)
“The ability of something to return to its original shape, after it has been pulled,stretched, pressed, or bent.”
Merriam-Webster
Resiliency
![Page 19: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/19.jpg)
![Page 20: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/20.jpg)
![Page 21: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/21.jpg)
![Page 22: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/22.jpg)
What about software systems?
![Page 23: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/23.jpg)
![Page 24: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/24.jpg)
![Page 25: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/25.jpg)
![Page 26: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/26.jpg)
![Page 27: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/27.jpg)
WHAT IF TOLD YOU
IT IS COMPLEX BUTNOT THAT COMPLICATED
![Page 28: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/28.jpg)
![Page 29: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/29.jpg)
![Page 30: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/30.jpg)
![Page 31: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/31.jpg)
Software Systems are Complex Systems
![Page 32: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/32.jpg)
“Complex systems run in degraded mode.”
“Complex systems run as broken systems.”
Richard Cook
![Page 33: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/33.jpg)
Asynchronous Communication
+
Eventual Consistency
Resilient Protocols
![Page 34: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/34.jpg)
Failures
Contained
Observed
ManagedReified as messages
![Page 35: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/35.jpg)
Message Driven
![Page 36: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/36.jpg)
Messages vs Events
SAVETHIS!
SOMEBODYLOGGED IN!
FactsTopic
Events
Past
AddressableSpecific
Messages
![Page 37: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/37.jpg)
REAL-TIME DATA INGESTION PLATFORM
![Page 38: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/38.jpg)
![Page 39: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/39.jpg)
Why Akka?
Reactive Elastic FaultTolerant
Load Management
Both up and out
LocationTransparency
![Page 40: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/40.jpg)
Akka Actors
Lightweight Reactive Asynchronous Resilient
![Page 41: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/41.jpg)
Challenges with Akka
Learning Curve Type Safety Debugging Dead
Letters
![Page 42: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/42.jpg)
Why Kafka?
Distributed Log
High Throughput Replicated Concurrency
![Page 43: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/43.jpg)
Kafka
Producer
Producer
Kafka Cluster
Broker 2
Topic 1Partition 1
Broker 1
Topic 1Partition 0
Broker 3
Topic 1Partition 3
Client
Client
Client
![Page 44: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/44.jpg)
Why Spark?
Fast! Unified Platform
FunctionalParadigm
Rich Library Set
ActiveCommunity
![Page 45: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/45.jpg)
PATTERNS AND ANTI-PATTERNS
![Page 46: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/46.jpg)
Ingestion
Hydra CoreIngestors
HTTP
Spark (Batch and Streaming)
Hydra CoreDispatchers
HTTP
RDBMS
HDFS
Conductors
Hydra CoreConductors
HTTP
Persistence :: Kafka
Hydra CorePersistence
HTTP
AKKARemoting
3
2
2
Hydra Topology
![Page 47: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/47.jpg)
GOOD PRACTICE:
DECENTRALIZE THE PROCESSING OF KEY TASKS
![Page 48: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/48.jpg)
HYDRA INGESTION MODULE
Actor Hierarchy
Supervision
Kafka Gateway
Message Protocol
![Page 49: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/49.jpg)
MESSAGE HANDLERS
![Page 50: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/50.jpg)
< META >
{ }/ingest
Coordinator
Registry
Handlers
Hydra Ingestion Flow
![Page 51: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/51.jpg)
Handler Registry
Monitors registered handlers for errors/stops
Broadcasts messages
Handler Lifecycle
![Page 52: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/52.jpg)
GOOD PRACTICE:
DESIGN AN INCREMENTAL COMMUNICATION PROTOCOL
![Page 53: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/53.jpg)
Hydra Ingestion Protocol
Publish
MESSAGEHANDLERS
Join
STOP
Validate IngestValid
Invalid<<Silence>>
![Page 54: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/54.jpg)
HEY GUYS!CHECK THIS
OUT!
HUH?! NICE!! BRINGIT!!
NAH…
Publish
JoinJoin
Hydra Ingestion Protocol: Publish
Handler Registry
Message handlers
![Page 55: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/55.jpg)
Hydra Ingestion Protocol: Validation
HOW DOESIT LOOK?
Validate
BAD!
Invalid
GOOD!
Valid
Ingestion Coordinator
Message handlers
![Page 56: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/56.jpg)
Hydra Ingestion Protocol: Invalid Message
Ingestion Coordinator
Error Reporter
GOT A BAD ONE
ReportError
Ingest
![Page 57: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/57.jpg)
for
ea
ch
ha
nd
ler
Hydra Ingestion Protocol: Ingest
SHIPIT!
Ingest
Encode Persist
![Page 58: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/58.jpg)
abstract class BaseMessageHandler extends Actor with ActorConfigSupport with ActorLogging with IngestionFlow with ProducerSupport with MessageHandler {
ingest { case Initialize => { //nothing required by default } case Publish(request) => { log.info(s"Publish message was not handled by ${self}. Will not join.") } case Validate(request) => { sender ! Validated }
case Ingest(request) => { log.warning("Ingest message was not handled by ${self}.") sender ! HandlerCompleted } case Shutdown => { //nothing required by default } case Heartbeat => { Health.get(self).getChecks } }}
![Page 59: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/59.jpg)
GOOD PRACTICE:
HIDE AN ELASTIC POOL OF RESOURCES BEHIND ITS OWNER
![Page 60: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/60.jpg)
Publisher SubscriberBack pressure
Less of this…
![Page 61: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/61.jpg)
RouterPublisher
Workers
More of this!
![Page 62: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/62.jpg)
akka { actor { deployment { /services-manager/handler_registry/segment_handler { router = round-robin-pool optimal-size-exploring-resizer { enabled = on action-interval = 5s downsize-after-underutilized-for = 2h } }
/services-manager/kafka_producer { router = round-robin-pool resizer { lower-bound = 5 upper-bound = 50 messages-per-resize = 500 } } } }}
![Page 63: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/63.jpg)
akka { actor { deployment { /services-manager/handler_registry/segment_handler { router = round-robin-pool optimal-size-exploring-resizer { enabled = on action-interval = 5s downsize-after-underutilized-for = 2h } } }
provider = "akka.cluster.ClusterRefActorProvider" }
cluster { seed-nodes = ["akka.tcp://[email protected]:2552","akka.tcp://[email protected]:2553"] }}
![Page 64: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/64.jpg)
GOOD PRACTICE:
USE SELF-DESCRIBING MESSAGES
![Page 65: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/65.jpg)
![Page 66: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/66.jpg)
![Page 67: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/67.jpg)
trait KafkaMessage[K, P] {
val timestamp = System.currentTimeMillis
def key: K
def payload: P
def retryOnFailure: Boolean = true}
case class JsonMessage(key: String, payload: JsonNode) extends KafkaMessage[String, JsonNode]
object JsonMessage { val mapper = new ObjectMapper()
def apply(key: String, json: String) = { val payload: JsonNode = mapper.readTree(json) new JsonMessage(key, payload) }}
case class AvroMessage(val schema: SchemaHolder, key: String, json: String) extends KafkaMessage[String, GenericRecord] {
def payload: GenericRecord = { val converter: JsonConverter[GenericRecord] = new JsonConverter[GenericRecord](schema.schema) converter.convert(json) }}
![Page 68: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/68.jpg)
GOOD PRACTICE:
PREFER BINARY DATA FORMATS FOR COMMUNICATION
![Page 69: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/69.jpg)
Why Avro?
Binary Format Space Efficient
Evolutionary Schemas
Automatic Tables
![Page 70: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/70.jpg)
GOOD PRACTICE:
DELEGATE AND SUPERVISE! REPEAT!
![Page 71: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/71.jpg)
![Page 72: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/72.jpg)
Error Kernel
![Page 73: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/73.jpg)
![Page 74: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/74.jpg)
Ingestion Actors: Coordinators
Supervises ingestion at the request level
Coordinates protocol flow
Reports errors and metrics
![Page 75: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/75.jpg)
GOOD PRACTICE:
LET IT CRASH
![Page 76: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/76.jpg)
Let it Crash
Components where full restarts are always ok
Transient failures are hard to find
Simplified failure model
![Page 77: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/77.jpg)
override val supervisorStrategy = OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) { case _: ActorInitializationException => akka.actor.SupervisorStrategy.Stop case _: FailedToSendMessageException => Restart case _: ProducerClosedException => Restart case _: NoBrokersForPartitionException => Escalate case _: KafkaException => Escalate case _: ConnectException => Escalate case _: Exception => Escalate }
val kafkaProducerSupervisor = BackoffSupervisor.props( Backoff.onFailure( kafkaProducerProps, childName = actorName[KafkaProducerActor], minBackoff = 3.seconds, maxBackoff = 30.seconds, randomFactor = 0.2 ))
![Page 78: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/78.jpg)
class KafkaProducerActor extends Actor with LoggingAdapter with ActorConfigSupport with NotificationSupport[KafkaMessage[Any, Any]] {
import KafkaProducerActor._
implicit val ec = context.dispatcher
override def preRestart(cause: Throwable, message: Option[Any]) = { //send it to itself again after the exponential delays, no Ack from Kafka message match { case Some(rp: RetryingProduce) => { notifyObservers(KafkaMessageNotDelivered(rp.msg)) val nextBackOff = rp.backOff.nextBackOff val retry = RetryingProduce(rp.topic, rp.msg) retry.backOff = nextBackOff context.system.scheduler.scheduleOnce(nextBackOff.waitTime, self, retry) } case Some(produce: Produce) => { notifyObservers(KafkaMessageNotDelivered(produce.msg)) if (produce.msg.retryOnFailure) { context.system.scheduler.scheduleOnce(initialDelay, self, RetryingProduce(produce.topic, produce.msg)) } } } }}
![Page 79: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/79.jpg)
Monitoring through Death Watches
![Page 80: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/80.jpg)
WHAT ABOUT SOME ANTI- PATTERNS?
![Page 81: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/81.jpg)
NOT SO GOOD PRACTICE:
BUILDING NANO SERVICES
![Page 82: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/82.jpg)
Ingestion
Hydra CoreIngestors
HTTP
Spark (Batch and Streaming)
Hydra CoreDispatchers
HTTP
RDBMS
HDFS
Conductors
Hydra CoreConductors
HTTP
Persistence :: Kafka
Hydra CorePersistence
HTTP
AKKARemoting
3
2
2
Hydra Topology
![Page 83: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/83.jpg)
NOT SO GOOD PRACTICE:
TREATING LOCATION TRANSPARENCY AS A FREE-FOR-ALL
![Page 84: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/84.jpg)
Guaranteed Delivery in Hydra
What does guaranteed delivery mean?
At most once semantics
Can be made stronger
![Page 85: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/85.jpg)
Akka Remoting
Peer-to-Peer Serialization Delivery Reliability Latency
![Page 86: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/86.jpg)
The Reliable Proxy Pattern
![Page 87: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/87.jpg)
@throws(classOf[Exception])override def init: Future[Boolean] = Future { val useProxy = config.getBoolean(“message.proxy”,false) val ingestorPath = config.getRequiredString("ingestor.path")
ingestionActor = if (useProxy) context.actorSelection(ingestorPath) else context.actorOf(ReliableIngestionProxy.props(ingestorPath))
val cHeaders = config.getOptionalList("headers") topic = config.getRequiredString("kafka.topic") headers = cHeaders match { case Some(ch) => List( ch.unwrapped.asScala.map { header => { val sh = header.toString.split(":") RawHeader(sh(0), sh(1)) } }: _* ) case None => List.empty[HttpHeader] } true}
![Page 88: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/88.jpg)
NOT SO GOOD PRACTICE:
NOT KEEPING MESSAGE PROTOCOL BOUND TO THEIR CONTEXTS
![Page 89: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/89.jpg)
object Messages {
case object ServiceStarted
case class RegisterHandler(info: ActorRef)
case class RegisteredHandler(name: String, handler: ActorRef)
case class RemoveHandler(path: ActorPath)
case object GetHandlers
case object InitiateIngestion extends HydraMessage
case class RequestCompleted(s: IngestionSummary) extends HydraMessage case class IngestionSummary(name:String)
case class Produce(topic: String, msg: KafkaMessage[_, _], ack: Option[ActorRef]) extends HydraMessage
case object HandlerTimeout extends HydraMessage
case class Validate(req: HydraRequest) extends HydraMessage
case class Validated(req: HydraRequest) extends HydraMessage
case class NotValid(req: HydraRequest, reason: String) extends HydraMessage
case object HandlingCompleted extends HydraMessage
case class Publish(request: HydraRequest)
case class Ingest(request: HydraRequest)
case class Join(r: HydraRequest) extends HydraMessage}
![Page 90: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/90.jpg)
class HandlerRegistry extends Actor with LoggingAdapter with ActorConfigSupport {
override def receive: Receive = { ... }
override val supervisorStrategy = OneForOneStrategy() { case e: Exception => { report(e) Restart } }}
object HandlerRegistry {
case class RegisterHandler(info: HandlerInfo)
case class RegisteredHandler(name: String, handler: ActorRef)
case class RemoveHandler(path: ActorPath)
case object GetHandlers}
![Page 91: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/91.jpg)
NOT SO GOOD PRACTICE:
DEVELOPING OVERLY CHATTY PROTOCOLS
![Page 92: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/92.jpg)
![Page 93: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/93.jpg)
![Page 94: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/94.jpg)
What’s next?
![Page 95: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/95.jpg)
Co
nd
uct
ors
We
bh
oo
ksWhat’s streaming into Hydra today?
![Page 96: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/96.jpg)
0
500
1000
1500
2000
2500
Dec-15 Jan-16 Jan-16 Jan-16 1-Feb 3/1/16
Average Ingestions Per Second
Requests
![Page 97: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/97.jpg)
9,730 lines of Scala code
Production Platform Since Jan 2016
C.I. through Jenkins and Salt
Some Facts
![Page 98: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/98.jpg)
![Page 99: Designing a reactive data platform: Challenges, patterns, and anti-patterns](https://reader036.fdocuments.in/reader036/viewer/2022062412/587984641a28ab6c358b617d/html5/thumbnails/99.jpg)
roarking
QUESTIONS?
Thank You!