Visual Analytics with Tableau & Couchbase: Couchbase Connect 2015
Couchbase at Cox Automotive – Couchbase Live New York 2015
Transcript of Couchbase at Cox Automotive – Couchbase Live New York 2015
![Page 1: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/1.jpg)
![Page 2: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/2.jpg)
Couchbase at Cox Automotive
![Page 3: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/3.jpg)
3
Who we are
![Page 4: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/4.jpg)
4
Who we are
![Page 5: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/5.jpg)
5
Who we are
![Page 6: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/6.jpg)
6
What we do at
![Page 7: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/7.jpg)
7
What we do at
![Page 8: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/8.jpg)
8
What we do at
![Page 9: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/9.jpg)
9
Our data changes quickly
![Page 10: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/10.jpg)
Couchbase at Cox AutomotiveKeeping pace with our data
![Page 11: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/11.jpg)
11
What sort of data do we store?
![Page 12: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/12.jpg)
12
What sort of data do we store?
![Page 13: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/13.jpg)
13
What sort of data do we store?
![Page 14: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/14.jpg)
14
How do we keep pace with our data?
![Page 15: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/15.jpg)
15
How do we keep pace with our data?
![Page 16: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/16.jpg)
16
How do we keep pace with our data?
![Page 17: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/17.jpg)
17
How do we keep pace with our data?
![Page 18: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/18.jpg)
18
How do we keep pace with our data?
Two weeks to get to production!
![Page 19: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/19.jpg)
Couchbase at Cox AutomotiveAccelerating our pace
![Page 20: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/20.jpg)
20
We churn through truckloads of data
![Page 21: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/21.jpg)
21
We churn through truckloads of data
![Page 22: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/22.jpg)
22
An API with Couchbase storage
![Page 23: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/23.jpg)
23
An API with Couchbase storage
![Page 24: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/24.jpg)
24
An API with Couchbase storage
![Page 25: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/25.jpg)
25
Implementation
API design features
• RESTful endpoints• Use of JSON format and immutable documents• Cached access to SQL Server look ups for configuration• Couchbase storage for speed and flexibility
![Page 26: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/26.jpg)
26
It’s simple to move our data into documents
SQL Server crashes when we try to list all our “staging” tables!
![Page 27: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/27.jpg)
27
It’s simple to move our data into documents{ "ImageURLs": [ "http://www.xhjsdfk.com/15234.jpg", "http://www.xhjsdfk.com/93784.jpg", "http://wwwxhjsdfk.com/13378.jpg" ], "Properties": { "BodyStyle": "2dr Car", "TransmissionSpeed": 6, "TransmissionTypeCode": "6-Speed Ma", "VehicleCondition": "New", "Year": 2021, }, "VIN": "1TAGP28135G148557" }, "Timestamp": "2015-10-05T13:38:46.2979624-04:00", "Version": 1}
JSON lends itself to flexible storage
There’s no schema
![Page 28: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/28.jpg)
28
It’s simple to move our data into documents
We coded an additive storage process.
Say we get this input first….
![Page 29: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/29.jpg)
29
It’s simple to move our data into documents
And then we get this data, for the same vehicle…
![Page 30: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/30.jpg)
30
It’s simple to move our data into documents
We produce a resulting document…
![Page 31: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/31.jpg)
31
It’s simple to move our data into documents
Now we’ve solved our
archiving problem andour rollback problem!
Version 1 Version 2
![Page 32: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/32.jpg)
32
How we got operational in two data centers
Now we use Couchbase as our database of record for this data,
we want resiliency with XDCR
![Page 33: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/33.jpg)
33
How we got operational in two data centers
Remember that TTL setting for the cached data?
Here’s some free advice
![Page 34: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/34.jpg)
Couchbase at Cox AutomotiveStaying at the front of the pack
![Page 35: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/35.jpg)
35
Wow; we’re maxed out again!
We have a fancy SQL Server*doing the heavy lifting
- 4 TB of primary data- Gigabytes of change per minute- 60 CPU cores!- 2TB RAM- 12TB Fusion IO cards!
*cluster of 3 machines
![Page 36: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/36.jpg)
36
Data model migration
Forward plans for SQL and NoSQL
OPERATIONAL CACHEDCOMPUTED
RAW DATA
![Page 37: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/37.jpg)
37
What functions should Couchbase serve?
• Easing the load for heavy-read objects from SQL Server• Applications where ‘eventually consistent’ is OK• Outbound ‘file’ storage for computed documents
• Novel queries against unstructured documents can leverage N1QL• When the query load is unpredictable, it can scale fast
![Page 38: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/38.jpg)
38
Future steps: How to migrate production data?
Our traditional approach
PRODUCTION STAGINGRESTORE
QA ENV.
C# APP
![Page 39: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/39.jpg)
39
Future steps: How to migrate production data?
With Couchbase 4.0
PRODUCTION STAGINGRESTORE
QA ENV.
XDCRFilter
![Page 40: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/40.jpg)
Couchbase at Cox AutomotiveConclusion
![Page 41: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/41.jpg)
41
How we began with Couchbase
“Start where you are.”- Chris Gardner
![Page 42: Couchbase at Cox Automotive – Couchbase Live New York 2015](https://reader036.fdocuments.in/reader036/viewer/2022081520/588a96181a28ab904b8b5f3f/html5/thumbnails/42.jpg)
Couchbase at Cox AutomotiveQuestions