Cassandra Codebase 2011
-
Upload
gdusbabek -
Category
Technology
-
view
4.848 -
download
1
description
Transcript of Cassandra Codebase 2011
![Page 1: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/1.jpg)
Codebase 2011
Getting to know the codebase
Gary Dusbabek@gdusbabek
![Page 2: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/2.jpg)
Questions?
![Page 3: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/3.jpg)
Outline
• How to contribute• Internals• Some thoughts
![Page 4: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/4.jpg)
How to Contribute
![Page 5: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/5.jpg)
How to Contribute
• http://wiki.apache.org/cassandra/HowToContribute
• JIRA: “lhf” label (Low hanging fruit)• Scratch your itch
![Page 6: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/6.jpg)
How to Contribute
• Run the tests• ant test• nosetests test/system/test_thrift_server.py
![Page 7: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/7.jpg)
How to Contribute
• http://wiki.apache.org/cassandra/CodeStyle
• Avoid:– Reformatting white space– Renaming things everywhere– Unrelated changes
![Page 8: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/8.jpg)
How to Contribute
• Use git• Attach patches– git format-patch as jira attachments.– Group them sensibly
![Page 9: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/9.jpg)
How to Contribute
• Someone will review your code– Usually a committer– Persistence helps
• Don’t get your feelings hurt– It usually takes a few rounds
![Page 11: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/11.jpg)
Internals
![Page 12: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/12.jpg)
Services
• Ring Operations (StorageService)• Storage Operations (StorageProxy)
![Page 13: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/13.jpg)
Startup Sequence
• bin/cassandra– Finds cassandra.in.sh• $CLASSPATH (mandatory)• $CASSANDRA_HOME• $CASSANDRA_CONF (mandatory)
– Executes $CASSANDRA_CONF/cassandra-env.sh• Sets heap sizes (gc tuning goes here!)
![Page 14: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/14.jpg)
o.a.c.thrift.CassandraDaemon
![Page 15: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/15.jpg)
AbstractCassandraDaemon
• ACD.setup():– Reads configuration: DatabaseDescriptor
– Loads schema: DD.loadSchemas()– Scrub directories– Initialize storage (keyspaces + CFs)– Commit log recovery: CL.recover()– StorageService.initServer() -> StorageService.joinTokenRing()
![Page 16: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/16.jpg)
Attn Tinkerers!
• Abstracted initialization of transport.• Handy if you’re experimenting with
transports/RPC• Just extend AbstractCassandraDaemon and make sure that class is started up via bin/cassandra.
![Page 17: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/17.jpg)
o.a.c.thrift.CassandraServer
• Implements thrift interface methods (the API).
• Start here when trying to understand the read/write path and RPC.
![Page 18: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/18.jpg)
Configuration
• DatabaseDescriptor– Side-effect of ACD.setup()
• Reads config settings from yaml• Defines system tables• Changes regularly
• I hate this code. Please fix it.
![Page 19: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/19.jpg)
Main Singletons
• StorageService• StorageProxy• MessagingService• CompactionManager• StageManager• MigrationManager
![Page 20: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/20.jpg)
Did you just say ‘Singletons?’
![Page 21: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/21.jpg)
Main Singletons
• StorageService• StorageProxy• MessagingService• CompactionManager• StageManager• MigrationManager
![Page 22: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/22.jpg)
JMX MBeans
• Tooling supplied by Mbeans• Anything that does
measureable/configurable work is tooled– Thread pools– Compaction– Hinted handoff– Streaming– Storage– Commit log
![Page 23: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/23.jpg)
StorageService
• initServer() -> joinTokenRing()– Starts gossip– Starts MessagingService– Negotiates bootstrap
• Many ring operations live here.• Repository of ring topology– TokenMetadata (quasi-singleton via
SS.tokenMetadata_)– Partitioner instance is also here
![Page 24: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/24.jpg)
MessagingService
• Verb handlers live here (initialized from SS).– Main event handlers, haven’t changed much.
• Socket listener– 2 threads per ring node
• Message gateway– emitted from MessageProducer impls– MS.sendRR()– MS.sendOneWay()– MS.receive()
• Messages are versioned now (0.8)– IncomingTCPConnection
![Page 25: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/25.jpg)
StorageProxy
• Top level of all read/write operations– Called from o.a.c.thrift.CassandraServer–Write path changed because of counters• Notion of WritePerformer
• Eventually to Table and ColumnFamilyStore
• Further, to SSTable and related classes.
![Page 26: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/26.jpg)
StageManager
• Fancy java ThreadPoolExecutor• SEDA:
http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf
• consumes callables from a queue.• Manages concurrency.• Hasn’t changed much.
![Page 27: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/27.jpg)
Adding API Methods
• Define method+structures in IDL– interface/cassandra.thrift
• Regenerate files– ant gen-thrift-java gen-thrift-py
• Implement stubs:– o.a.c.thrift.CassandraServer
• Create a system test– tests/system/test_thrift_server.py
![Page 28: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/28.jpg)
Reading
• Socket->CassandraServer– Permissions– Request validation–Marshalling
• ReadCommands created in CS.multigetSliceInternal, passed to StorageProxy– 1 per key
![Page 29: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/29.jpg)
Reading
• StorageProxy.read(), fetchRows()
• For each ReadCommand– Determine endpoints– Local & remote
branches
![Page 30: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/30.jpg)
Reading
• StorageProxy local• READ stage executes a LocalReadRunnable– True read vs digest– Table, ColumnFamilyStore– CFS.getTopLevelColumns
• Make QueryFilter• Query Memtables• Query SSTables• Coalesce in iterators
![Page 31: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/31.jpg)
Reading
• StorageProxy remote– read command– Response handler– Send to remote nodes
• Read repair happens in SP.fetchRows().
![Page 32: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/32.jpg)
Writing
• CS.doInsert()– Marshalling, creates RMs
• StorageProxy – local/remote branch– SP.sendToHintedEndpoints()
• RowMutation– one Key per (several
CFs)
• ColumnFamily– Collection of column
modifications
![Page 33: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/33.jpg)
Writing
• RM.apply->Table.apply–Write to CL– Iterate over RM CFs– CFS.apply()• Overwrites results
on pre-existing column families
![Page 34: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/34.jpg)
Writing
• RM is serialized into a Message and sent to other nodes–Waits for ACKs
depending on CL
![Page 36: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/36.jpg)
Challenges
• To have an in-depth understanding of everything.–Hard for hobbyist/part-timers–Outside of Datastax, little support
for full-timers–Still changing fast• Keeping up
![Page 37: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/37.jpg)
Challenge: Lines of Code
• 0.4 (Sep 2009)– 52 kloc
• 0.5 (Jan 2010)– 59 kloc
• 0.6 (Apr 2010)– 73 kloc
• 0.7 (Jan 2011)– 122 kloc
• 0.8 (Jun 2011)– 146 kloc
• Trunk (yesterday)– 149 kloc
Average:4,500 lines per month
![Page 38: Cassandra Codebase 2011](https://reader033.fdocuments.in/reader033/viewer/2022061202/547c7036b4af9fdb728b4599/html5/thumbnails/38.jpg)
Challenges
• Codewise Growing pains– Software maturity– Decisions made early on