Agile Document Models & Data Structures

©2016CouchbaseInc.

AgileDocumentModels&DataStructures

1

©2016CouchbaseInc.©2016CouchbaseInc.

SpeakingYourLanguage

•  Topicsfortoday:•  Datastructures-tieintonativelanguagecollectioninterfaces•  Sub-document-lowerlevelaccesswithfocusedpower•  DatamodelingwithCouchbase•  Session:“PickingtherightAPIfortherightjob”

•  SDKGoal:complexdataaccessmadeeasy•  Morethanjustadocumentstorage/retrievalsystem•  TightSDKintegrationiskey•  Consistent,transparentdeveloperexperienceacrosslanguages

2

©2016CouchbaseInc. 3

DataStructuresAPI


CouchbaseSDKDataStructuresAPI

•  TargetSDKreleasealongwith4.6•  Buildsonawesomenessofsub-documentAPI

•  Simplifiedaccesswithouttouchingwholedocument

•  MakeJSONdatatypestransparent

•  NativeintegrationofMap,List,Set,Queues…•  JavaCollectionsFramework•  .NETSystem.Collections•  Python,Node.js,Go

4


TypicalDocumentDataAccess

JSONDocCB JSON

ObjectSDK Collec?onsFrameworkApp

5


SimplifiedDataStructureAccess

JSONDocCB Collec?ons

FrameworkSDKDS

6

App

“user1”: {“name”:... , “address”:.. ,

“favs”: [...]}, “user2”:{“name” ,

“address” ..., ”favs”: [...]},

for (String f : favs) {}

“user1”: {“name”:... , “address”:.. , “favs”: [...]},

“user2”:{“name” , “address” ..., ”favs”: [...]},


TargetedCollectionUpdates

ItemFrom

Collec?onAppSub-doc

Update CB

7

MapAdd(“user1”,”favs”, “newfav”)

“user1”: {“name”:... , “address”:.. , “favs”: [...]},

“user2”:{“name” , “address” ..., ”favs”: [...]},


TheFourDataStructures…

Structure JSONType JSONExampleLists-  Append,prepend,insert-  Size/count

JSONArray:[…,...] [1,2,“abc”]

Maps-  Add/removebykey-  Size/count

JSONObject:{“key”:“value”}

{“name”:“value”}

Sets-  Specializedadd/remove-  Uniquevalues-  Size/count

JSONArray:[…,...] [1,3,6,8]

Queue-  Firstin–firstout-  Pop–retrieve/remove-  Size/count

JSONArray:[…,...]

[“task1”,“task2”,“task3”]remove1...[“task2”,“task3”,“task4”]

8


ConsistentAccessAcrossLanguages

Func:onsLists ListGet ListPush ListShift ListDelete ListSet ListSize

>namesList = bucket.ListGet(“key”)>print namesList[‘name1’,’name2’,’name3’]

Maps MapGet MapRemove MapSize MapSetSets SetAdd SetExists SetSize SetRemoveQueue QueuePush QueuePop QueueSize QueueRemove

9

•  Idiomatic-vs-functional•  JavaCollectionsFramework•  .NETSystem.Collections•  Aswellasfunctionalapproach

*Experimentalfeaturesalert:mayadd/removetothislist–feedbackwelcome!


ConsistentAccessAcrossLanguages

10

Collec:onsApproach

Lists List<String> namesList = new CouchbaseArrayList<String>("key", bucket);for (String name : namesList) { … }

Maps var namesDict = new CouchbaseDictionary<string, Poco>(_bucket, “key”); namesDict.Add(“newkey1”, new Poco { Name = “poco1” });

Sets var namesSet = new CouchbaseSet<Poco>(_bucket, "pocos");namesSet.Add(new Poco { Key = "poco1", Name = "Poco-pica" });namesSet.Remove(new Poco {Key = "poco1", Name = "Poco-pica"});foreach(var poco in namesSet){ … }

Queue var namesQueue = new CouchbaseQueue<Poco>(_bucket, key);namesQueue.Enqueue(new Poco { Name = "pcoco1" });var item = namesQueue.Peek();

•  Supportforadvancedcapabilitiesofcollectionframeworks


Sub-DocumentAPI


Sub-DocumentAPI

“Thesub-documentAPIenablesyoutoaccesspartsofJSONdocuments(sub-documents)efficientlywithoutrequiringthetransferoftheentiredocumentoverthenetwork.

ThisimprovesperformanceandbringsbetterefficiencytothenetworkIOpath,especiallywhenworkingwithlargeJSONdocuments.”

•  Firstreleasedin4.5,supportcrossSDK•  Efficientdocumentlookup,insert&update

•  Powerfullowerlevelcontrol,focusingonparticularelements

•  Keepworkonserver

•  Twomethodsavailable–lookupandmutate/change

12


DiggingBelowDataStructures

DataStructuresAPI Sub-DocumentAPI

MapGet(key, mapkey) LookupIn(key).get(mapkey)

MapRemove(key, mapkey) MutateIn(key).remove(mapkey)

MapSet(key, mapkey, value, createMap)

MutateIn(key).(mapkey, value, create_doc=createMap)

13


Sub-DocumentAPI

14

Opera:onsLookupIn LookupIn(key, operation(path))

Get Exists Execute

MutateIn MutateIn(key, operation(path, value))

Counter Insert Remove Replace Upsert Execute arrayAddunique arrayAppend arrayInsert arrayPrepend

ChainingOpera:ons

MutateIn(key, operation(path, value), operation(path, value), operation(path, value))

Returns SubdocResult<rc=0x0, key='map1', cas=0x14b6458980042, specs=(Spec<GET, 'subkey1'>, Spec<EXISTS, 'subkey1'>), results=[(0, u'subvalue1'), (0, None)]>


SampleSub-DocumentLookup

15

LookupIn(key, operation(path))

LookupIn(‘copilotmark’) .get(‘phones.number') .execute();

LookupIn(‘copilotmark’) .exists(‘phones’) .get(‘phones.number') .get(‘gender’) .execute();

SubdocResult<rc=0x0, key=’copilotmark', cas=0x14b6458980042, specs=(Spec<EXISTS, ‘phones’>, <GET, ’phones.number'>, <GET, ‘gender’), results=[(0, None,), (0, ’212-771-1834’), (0, u’male')]>


SampleSub-DocumentChange

16

MutateIn(key, path, value)MutateIn(‘copilotmark’) .replace(‘phones.number’,

‘212-787-2212’) .upsert(‘nickname’, ‘Freddie’) .execute()


DataModelingforCouchbaseServer


WhatisDataModeling?

18

•  Adatamodelisaconceptualrepresentationofthedatastructuresthatarerequiredbyadatabase

•  Thedatastructuresincludethedataobjects,theassociationsbetweendataobjects,andtheruleswhichgovernoperationsontheobjects.


DataModelingApproaches

19

NoSQLRelaxedNormaliza?onschemaimpliedbystructurefieldsmaybeempty,duplicate,ormissing

Rela:onalRequiredNormaliza?on

schemaenforcedbydbsamefieldsinallrecords

•  Minimizedatainconsistencies(oneitem=oneloca?on)•  Reducedupdatecost(noduplicateddata)•  Preservestorageresources

•  Op?mizedtoplanned/actualaccesspagerns•  Flexiblywithsoiwarearchitecture•  Supportsclusteredarchitecture•  Reducedserveroverhead


ModelingCouchbaseDocuments

20

•  CouchbaseServerisadocumentdatabase•  DataisstoredinJSONdocuments,notintables

•  Relationaldatabasesrelyonanexplicitpre-definedschematodescribethestructureofdata

•  JSONdocumentsareself-describing


WhatandWhyJSON?

21

•  WhatisJSON?–  Lightweightdatainterchangeformat–  BasedonJavaScript–  Programminglanguageindependent–  Fieldnamesmustbeunique

•  WhyJSON?–  Schemaflexibility–  Lessverbose–  CanrepresentObjectsandArrays

(includingnesteddocuments)

ThereisNOIMPEDENCEMISMATCHbetweenaJSONDocumentandaJavaObject


JSONDesignChoices

22

•  CouchbaseServerneitherenforcesnorvalidatesforanyparticulardocumentstructure

•  ChoicesthatimpactJSONdocumentdesign:–  SingleRootAttributes–  Objectsvs.Arrays–  ArrayElementTypes–  TimestampFormats–  PropertyNames–  EmptyandNullPropertyValues–  JSONSchema


RootAttributesvs.EmbeddedAttributes

23

•  Thechoiceofhavingasinglerootattributeorthe“type”attributeembedded.



24

•  Accessingthedocumentwitharootattribute

SELECTtrack.*FROMcouchmusic



25

•  Accessingthedocumentwiththe“type”attribute

SELECT*FROMcouchmusic

WHERE

type=‘track’


Objectsvs.Arrays

26

•  Thechoiceofhavinganobjecttype,oranarraytype


Objectsvs.Arrays

27

•  Howwouldtheobjectlooklike?

class UserProfile{

Phone phones;

}

class Phone{

String cell;

String landline;

}


Objectsvs.Arrays

28

•  Howwouldtheobjectlooklike?

class UserProfile{

List<Phone> phones;

}

class Phone{

String number;

String type;

}


ArrayElementTypes

Arrayofstrings

Arrayofobjects

29

•  Arrayelementscanbesimpletypes,objectsorarrays:


ArrayElementTypes

Arrayofstrings

30


class Playlist{

List<String> tracks;

}

...

String trackId = tracks.get(1);

JsonDocument trackDocument =

bucket.get(trackId)

Mul:pleget()callstoretrievethedocument.Worthit?


ArrayElementTypes

31


class Playlist{

List<Track> tracks;

}

...

myPlaylist.getTracks()

.get(1).getArtistName();

LimitedDenormaliza:on:commonlyneededdata(e.g.,?tle)inlocalobject,detailavailableinreferencedforeigndocument


TimestampFormats

Arrayof?mecomponents

String(ISO8601)

Number(Unixstyle)(Epoch)

•  Workinganddealingwithtimestampshasbeenchallengingeversince

•  Whenstoringtimestamps,youhaveatleast3options:

16


ObservedPracticeswithTimestampFormats

•  StoringasEpochwillhelpyoutoeasilysortthedocuments•  Ifyouwantedthedocumentstobesortedintheorderoftheir“lastupdate”time•  SELECT * FROM couchmusic WHERE type = ‘track’

ORDER BY updates DESC

•  Storingdateasarrayformathelps•  Togrouping

16


TakingAdvantageofStoringDateasanArray

•  Groupoptionscanbespecifiedtocontroltheexecutionoftheview

•  Thegroupandgroup_leveloptionsareonlyusefulwhenaReducefunctionhasbeendefinedinthecorrespondingView

•  Thegroup_leveloption,usedwhenthekeyisanArray,determineshowmanyelementsofthekeyareusedwhenaggregatingtheresults.

16


Example of View group_level = 1

Key Value

[2014] 36

[2015] 20ExecuteReduce

Key Value

[2014,11,29,18,49,36] 3

[2014,12,03,20,11,26] 5

[2014,12,03,23,37,21] 2

[2014,12,06,10,12,19] 8

[2014,12,09,05,01,26] 3

[2014,12,18,01,04,30] 11

[2014,12,26,18,34,44] 4

[2015,01,03,16,48,32] 7

[2015,01,03,20,20,06] 5

[2015,01,15,08,17,28] 8

Copyright©2015Couchbase,Inc. 35

•  ForthedatabelowwithReducefunctiondefinedas_sumandgroup_level=1



Key Value

[2014,11] 3

[2014,12] 33

[2015,01] 20

Key Value

[2014,11,29,18,49,36] 3

[2014,12,03,20,11,26] 5

[2014,12,03,23,37,21] 2

[2014,12,06,10,12,19] 8

[2014,12,09,05,01,26] 3

[2014,12,18,01,04,30] 11

[2014,12,26,18,34,44] 4

[2015,01,03,16,48,32] 7

[2015,01,03,20,20,06] 5

[2015,01,15,08,17,28] 8



ExecuteReduce



Key Value

[2014,11,29,18,49,36] 3

[2014,12,03,20,11,26] 5

[2014,12,03,23,37,21] 2

[2014,12,06,10,12,19] 8

[2014,12,09,05,01,26] 3

[2014,12,18,01,04,30] 11

[2014,12,26,18,34,44] 4

[2015,01,03,16,48,32] 7

[2015,01,03,20,20,06] 5

[2015,01,15,08,17,28] 8

Key Value

[2014,11,29] 3

[2014,12,03] 7

[2015,12,06] 8

[2015,12,09] 3

[2015,12,18] 11

[2015,12,26] 4

[2014,01,03] 12

[2014,01,15] 8



ExecuteReduce


EmptyandNullPropertyValues

38

•  KeepinmindthatJSONsupportsoptionalproperties

•  Ifapropertyhasanullvalue,considerdroppingitfromtheJSON,unlessthere'sagoodreasonnotto

•  N1QLmakesiteasytotestformissingornullpropertyvalues

•  Besureyourapplicationcodehandlesthecasewhereapropertyvalueismissing

SELECT * FROM couchmusic1 WHERE userprofile.address IS NULL; SELECT * FROM couchmusic1 WHERE userprofile.gender IS MISSING;


Empty,NullandMissingPropertyValues

39

{ countryCode: “UK”, currencyCode: “GBP”, region: “Europe” }

{ countryCode: “UK”, currencyCode: “GBP”, region: “” }

WHERE region IS NOT MISSING, IS NOT NULL, IS VALUED

WHERE region IS NOT MISSING, IS NOT NULL, IS NOT VALUED

{ countryCode: “UK”, currencyCode: “GBP” }

{ countryCode: “UK”, currencyCode: “GBP”, region: null }

WHERE region IS MISSING WHERE region IS NULL


JSONSchema

40

•  CouchbaseServerpaysabsolutelynoattentiontotheshapeofyourJSONdocumentssolongastheyarewell-formed

•  TherearetimeswhenitisusefultovalidatethataJSONdocumentconformstosomeexpectedshape

•  JSONSchemaisaJSON-basedformatfordefiningthestructureofJSONdata

•  Thereareimplementationsformostpopularprogramminglanguages

•  Learnmorehere:http://json-schema.org


ExampleofJSONSchema

41


ExampleofJSONSchema–TypeSpecification

Availabletypespecifica?onsinclude:•  array•  boolean•  integer•  number•  object•  string•  enum

42


Typespecificvalida?onsinclude:•  minimum•  maximum•  minLength•  maxLength•  format•  pagern

43

ExampleofJSONSchema–TypeSpecificValidation


ExampleofJSONSchema–RequiredProperties

Requiredproper?escanbespecifiedforeachobject

44


ExampleofJSONSchema–AdditionalProperties

Addi?onalproper?escanbedisabled

45


DataNesting(akaDenormalization)

46

•  Asyouknow,relationaldatabasedesignpromotesseparatingdatausingnormalization,whichdoesn’tscale

•  ForNoSQLsystems,weoftenavoidnormalizationsothatwecanscale

•  Nestingallowsrelatedobjectstobeorganizedintoahierarchicaltreestructurewhereyoucanhavemultiplelevelsofgrouping

•  Ruleofthumbistonestnomorethan3levelsdeepunlessthereisaverygoodreasontodoso

•  Youwilloftenwanttoincludeatimestampinthenesteddata


Example#1ofDataNesting

•  Playlistwithownerattributecontainingusernameofcorrespondinguserprofile

47

DocumentKey:copilotmarks61569



•  Playlistwithownerattributecontainingasubsetofthecorrespondinguserprofile

48

*Notetheinclusionoftheupdatedagribute



•  PlaylistwithtracksattributecontaininganarrayoftrackIDs

49



•  Playlistwithtracksattributecontaininganarrayoftrackobjects

50

*Notetheinclusionoftheupdatedagribute


KeyDesign


ChoiceswithJSONKeyDesign

52

•  Akeyformedofattributesthatexistintherealworld:–  Phonenumbers–  Usernames–  Socialsecuritynumbers–  Accountnumbers–  SKU,UPCorQRcodes–  DeviceIDs

•  Oftenthefirstchoicefordocumentkeys

•  Becarefulwhenworkingwithanypersonallyidentifiableinformation(PII),sensitivepersonalinformation(SPI)orprotectedhealthinformation(PHI)


SurrogateKeys

53

•  Weoftenusesurrogatekeyswhennoobviousnaturalkeyexist

•  Theyarenotderivedfromapplicationdata

•  Theycanbegeneratedvalues–  3305311F4A0FAAFEABD001D324906748B18FB24A(SHA-1)–  003C6F65-641A-4CGA-8E5E-41C947086CAE(UUID)

•  Theycanbesequentialnumbers(oftenimplementedusingtheCounterfeatureofCouchbaseServer)–  456789,456790,456791,…


KeyValuePatterns

•  CommonpracticeforusersofCouchbaseServertofollowpatternsforformattingkeyvaluesbyusingsymbolssuchassingleordoublecolons

•  DocType::ID–  userprofile::fredsmith79–  playlist::003c6f65-641a-4c9a-8e5e-41c947086cae

•  AppName::DocType::ID–  couchmusic::userprofile::fredsmith79

54

EnablesMul:-Tenency

–  pizza::user::101–  Pizza::user::102

–  burger::user::101–  burger::user::102


LookupKeyPattern

55

•  ThepurposeoftheLookupKeyPatternistoallowmultiplewaystoreachthesamedata,essentiallyasecondaryindex

•  Forexample,wewanttolookupaUserprofilebytheiremailaddressinsteadoftheirID

•  Toaccomplishthis,wecreateanothersmalldocumentthatreferstotheUserprofiledocumentweareinterestedin

•  Implementingthispatternisstraightforward,justcreateanadditionaldocumentcontainingasinglepropertythatstoresthekeytotheprimarydocument

•  WiththeintroductionofN1QL,thispatternwillbelesscommonlyused


LookupKeyPattern

56

userprofile::copilotmarks61569 [email protected]

JSON

[email protected]

•  LookupdocumentcanbeJsonDocumentorStringDocument


Trade-offsinDataModeling


MakingToughChoices

58

•  Wemustalsomaketrade-offsindatamodeling:–  Documentsize–  Atomicity–  Complexity–  Speed


DocumentSize

59

•  CouchbaseServersupportsdocumentsupto20Mb

•  Largerdocumentstakemorediskspace,moretimetotransferacrossthenetworkandmoretimetoserialize/deserialize

•  Ifyouaredealingwithdocumentsthatarepotentiallylarge(greaterthan1Mb),youmusttestthoroughlytofindoutifspeedofaccessisadequateasyouscale.Ifnot,youwillneedtobreakupthedocumentintosmallerones.

•  Youmayneedtolimitthenumberofdependentchildobjectsyouembed


Atomicity

60

•  AtomicityinCouchbaseServerisatthedocumentlevel

•  CouchbaseServerdoesnotsupporttransactions

•  Theycanbesimulatedifyouarewillingtowriteandmaintainadditionalcodetoimplementthem(generallynotrecommended)

•  Ifyouabsolutelyneedchangestobeatomic,theywillhavetobepartofthesamedocument

•  ThemaximumdocumentsizeforCouchbaseServermaylimithowmuchdatayoucanstoreinasingledocument


Complexity

61

•  Complexityaffectseveryareaofsoftwaresystemsincludingdatamodeling

•  Thecomplexityofqueries(N1QL)

•  Thecomplexityofcodeforupdatingmultiplecopiesofthesamedata


Speed

62

•  Asitrelatestodatamodeling,speedofaccessiscritical

•  WhenusingN1QLtoaccessdata,keepinmindthatquerybydocumentkeyisfastestandquerybysecondaryindexisusuallymuchslower

•  Ifimplementinganinteractiveusecase,youwillwanttoavoidusingJOINs

•  Youcanusedataduplicationtoimprovethespeedofaccessingrelateddataandthustradeimprovedspeedforgreatercomplexityandlargerdocumentsize

•  KeepinmindthatCouchbaseViewscanbeusedwhenuptothesecondaccuracyisnotrequired


Remember

63

SDKget()isfasterthan(getbykey)

N1QLwithMOIisfasterthan

N1QLwithGSIisfasterthan

Modelyoudocumentkey,suchthatyoudocumentcanberetrievedwiththekey,ifpossible,thanaN1QLquery


Embedvs.Refer

64

•  Alloftheprevioustrade-offsareusuallyrolledintoasingledecision–whethertoembedorrefer

•  Whentoembed:–  Readsgreatlyoutnumberwrites–  You'recomfortablewiththeslimriskofinconsistentdataacrossthemultiplecopies–  You'reoptimizingforspeedofaccess

•  Whentorefer:–  Consistencyofthedataisapriority–  Youwanttoensureyourcacheisusedefficiently–  Theembeddedversionwouldbetoolargeorcomplex


NextSteps

•  Flexibledataaccessiskeytosolutionsusingdocumentstores

•  JoinusfordiscussiononForumsordiscusswithourexpertshere•  https://forums.couchbase.com •  https://developer.couchbase.com/server

65


GetTrainedonCouchbasehttp://training.couchbase.com

http://training.couchbase.com/online

CS300:CouchbaseNoSQLServerAdministrationCD220:DevelopingCouchbaseNoSQLApplications

CD210:CouchbaseNoSQLDataModeling,Querying,andTuningUsingN1QLCD257:DevelopingCouchbaseMobileNoSQLApplications


TylerMitchellSeniorProductManager,SDK

[email protected]@1tylermitchell

ClarenceJMTauro,Ph.D.SeniorInstructor

[email protected]@javapsyche

Agile Document Models & Data Structures

Documents

Transcript of Agile Document Models & Data Structures