Building Resilient and Evolutionary Data Microservices
-
Upload
spring-by-pivotal -
Category
Technology
-
view
911 -
download
0
Transcript of Building Resilient and Evolutionary Data Microservices
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Building Resilient and Evolutionary Data Microservices
By Vinicius Carvalho@vccarvalhoviniciusccarvalho
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Agenda• Data microservices• Why you should care about schema evolution• It’s all about the format• Introducing Schema evolution for Spring Cloud Stream• Demo
2
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data microservices
3
Self contained
Loosecoupled
EventDriven
Dataaware
Independent
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data microservices
4
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data pipelines
5
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data microservices in action: Context maps
6
Sam Newman’s Building Microservices
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data microservices in action: CQRS
7
MSDN CQRS MSDN CQRS Journey ebook
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Learn more
8
Cloud Native Streaming and Event-Driven MicroservicesWed 8/3 - 4:20 - 5:30 PM Starvine 2
Orchestrate All the Things! with Spring Cloud Data FlowThu 8/4 - 11:10 - 12:15 Starvine 2
Architecting for Cloud Native Data: Data Microservices Done Right Using Spring CloudWed 8/3 2:00 - 3:10 PM Starvine 2
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Evolutionary?
9
What happens when data evolve?
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema Evolution
10
“The problem of evolving a data schema to adapt it to a change in modeled reality”
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Backward compatibility
11
V1
V2
• A reader using a newer schema version can read from writers of an older version
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Forward compatibility
12
V1
V2
• A reader using an older schema version can read from writers of a newer version
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Request / Response paradigm
13
It’s all about a handshake
client<V1>
RecommendationService
GET /v1/…
<V2>Recommendation
Serviceclient
GET /v2/…
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Messaging paradigm
14
<V1>Recommendation
Service
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Messaging paradigm
15
<V1>Recommendation
Service
<V2>Recommendation
Service
if msg.headers[version] == “v1” {
}else if …
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Data Microservices
16
But I thought microservices were all about smart endpointsand dumb pipes
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Smart endpoints• Our goal is to make our endpoints
capable of adapting to changes on the model without requiring changes on the system.
17
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
How can we achieve that?
18
We have the tools We just need to evolve on how to use it
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
It all depends on the right format
19
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Choosing the right format : Features
20
✓Compact✓Strongly typed✓Adaptable✓Versioned✓Centralized
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Choosing a format
21
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Payload size
22
0
350
700
1050
1400
Payload Size (bytes)
1355
763
502
285243216
AvroProto3KryoCSVJSONXML
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Performance
23
Average marshaling time in nanos
Read Write
3,4833,433
1,3001,333
AvroJSON
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Benchmark : because we love one
24
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Bottom line
25
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Why Avro• Binary and compact format• Mature and with active community. Heavily used on hadoop ecosystem• Schemas are a first class citizen on it• Out of the box support for Schema evolution• Support for GenericRecords : Easy integration with SpEL• Cross language implementations for clients
26
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema evolution support on SCSt
27
API Level 16
API Level 21 Registry•Stores schemas•Tests compatibility•Tracks utilization
1. Producers registers schemas2. SCSt broadcast location in header
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema evolution support on SCSt
28
API Level 16
API Level 21 Registry
1. Consumers fetch message2. Query registry for schema3. If version is different apply resolution
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Starting the Server
29
@SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema Registry Server
30
• POST / : Register a new schema, returning it’s version and a location header
• GET /{subject}/{format}/{version} : Returns a schema definition
• GET /{subject}/{format} : Returns all versions of a schema for a given format
• POST /{subject}/{format}/test : Compatibility test against all registered versions
• GET /{subject}/{format}/usage : List apps using all versions of a schema
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Enabling Schema registration
31
@SpringBootApplication @EnableSchemaRegistryClient public class ProducerApplication { public static void main(String[] args) { SpringApplication .run(ProducerApplication.class,args); } }
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Enabling Schema registration - Producer
32
spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: output: destination: sensor-topic contentType: "application/*+avro"
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Enabling Schema registration - Consumer
33
spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: input: destination: sensor-topic
Content type is inferred from headers
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema evolution example
34
#v1 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"temperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]}
] }
#v2 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"internalTemperature", "type":"float", "default":0.0, "aliases":["temperature"]}, {"name":"externalTemperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]} ]
}
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Schema Support on Spring Cloud Stream
35
- DEMO -
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
References
36
• Martin Kleppmann Schema Evolution in avro, thrift and protobufers: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
• http://dataintensive.net/ - Martin Kleppmann
• The CQRS Journey: https://msdn.microsoft.com/en-us/library/jj554200.aspx
• Oracle Datastore schema evolution : https://docs.oracle.com/cd/NOSQL/html/GettingStartedGuide/schemaevolution.html
• Building Microservices by Sam Newman: http://samnewman.io/books/building_microservices/
• Apache Avro: https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
• https://github.com/viniciusccarvalho/schema-evolution-samples