Go’ing to the Next Level, Lessons from Go at Couchbase and Introducing the Go SDK: Couchbase...

GO’ING TO THE NEXT LEVEL, LESSONS FROM GO AT COUCHBASE AND INTRODUCING THE GO SDK

Marty Schoch and Matt Ingenthron

Lessons from Goat Couchbase

Lessons from Go at Couchbase

©2015 Couchbase Inc. 4

Couchbase Labs

2011 2 engineers 0 rules


What if we used Go?

Easy concurrency Easy HTTP Server Easy HTTP Client Easy JSON Rich Standard Library


Timeline

January 2012go-couchbase

2012

2014

September 2012sync getaway

September 2012cbfs

October 2012

tuq -> tuqtng -> n1ql -> sql for documents

November 2012cbgb

January 2013cbugg

February 2013

Godu, vbmap, and other utils

August 2013

Secondary indexing April 2014

bleve

August 2014goxdcr

October 2014cbft

December 2014gocb


Why do developers like Go?

Excellent tooling CPU/memory profilers Race detector Integrated Unit Tests

Channels for synchronization as well as communication

Satisfy interface without importing its package


What don’t developers like about Go?

No good debugger gdb works in some cases not others Others still immature

error as interface is powerful construct … but too often its fmt.Errorf(“…”) … which leads to string matching later Define constants so that errors can be differentiated and

handled accordingly

Garbage Collection


Don’t Make Garbage


Sounds Simple

Be mindful of allocations Be mindful of operations that copy data

string <-> []byte conversions Many standard library functions operating on []byte But, once you avoid making copies you need clear notion

of ownership Re-use already allocated objects when possible

Include Reset() methods in your API sync.Pool offers solution when objects are

reusable and have clear start/end lifecycle


Do I really have a GC problem?

Review CPU Profiles



Graph GC Pause Time (exposed by expvar)



Visualize Garbage Collector Details Using gcvis by Dave Cheney


Is this variable on the stack or the heap?

Have the compiler tell you. Build with: go build -gcflags=-m

./main.go:54: map[string]interface {} literal escapes to heap./main.go:112: (*matchPhraseQuery).Validate q does not escape


Solutions for Advanced Users

Slab allocation See github.com/couchbase/go-slab

Off-heap allocation Lots of pointers, even if largely static, may cause

unreasonable GC Large maps, tree structures with child pointers, etc GC has to check pointers in map, even though you never

plan on freeing them Several new projects in this area…


Other Memory Surprises?


Sync Gateway – Many Concurrent Users

Observation: often 20GB+ memory usage Action: Review heap profile

(pprof) top25Total: 3972.5 MB 3875.0 97.5% 97.5% 3875.0 97.5% compress/flate.(*compressor).initDeflate 19.0 0.5% 98.0% 19.5 0.5% github.com/couchbaselabs/sync_gateway/db.(*Database).MultiChangesFeed 12.5 0.3% 98.3% 12.5 0.3% net/http.newBufioWriterSize 10.5 0.3% 98.6% 10.5 0.3% bufio.NewReaderSize 10.0 0.3% 98.9% 10.0 0.3% compress/flate.newHuffmanBitWriter 7.0 0.2% 99.0% 7.0 0.2% newdefer 4.5 0.1% 99.1% 4.5 0.1% github.com/couchbaselabs/sync_gateway/channels.readString 3.0 0.1% 99.2% 3.0 0.1% net/textproto.(*Reader).ReadMIMEHeader 2.0 0.1% 99.3% 2.0 0.1% github.com/couchbaselabs/sync_gateway/auth.func·001 2.0 0.1% 99.3% 2.0 0.1% reflect.mapassign 1.5 0.0% 99.4% 1.5 0.0% github.com/couchbaselabs/sync_gateway/channels.TimedSet.Copy 1.5 0.0% 99.4% 1.5 0.0% net/http.Header.clone 1.5 0.0% 99.4% 1.5 0.0% net/textproto.MIMEHeader.Set 1.5 0.0% 99.5% 1.5 0.0% net/url.parseQuery 1.5 0.0% 99.5% 1.5 0.0% runtime.malg 1.0 0.0% 99.5% 1.0 0.0% compress/flate.(*huffmanEncoder).generate 1.0 0.0% 99.6% 1.0 0.0% concatstring 1.0 0.0% 99.6% 1.0 0.0% encoding/json.(*decodeState).objectInterface 1.0 0.0% 99.6% 1.0 0.0% github.com/couchbaselabs/sync_gateway/channels.(*ChangeLog).TruncateTo 1.0 0.0% 99.6% 5.5 0.1% github.com/couchbaselabs/sync_gateway/channels.decodeChangeLog 1.0 0.0% 99.7% 1.0 0.0% github.com/gorilla/context.Set 1.0 0.0% 99.7% 1.0 0.0% github.com/gorilla/mux.(*routeRegexpGroup).setMatch 1.0 0.0% 99.7% 4.5 0.1% net/http.ReadRequest 0.5 0.0% 99.7% 3885.5 97.8% compress/flate.NewWriter 0.5 0.0% 99.7% 0.5 0.0% compress/gzip.NewWriterLevel

http://github.com/couchbaselabs/sync_gateway/db

http://github.com/couchbaselabs/sync_gateway/channels.readString

http://github.com/couchbaselabs/sync_gateway/auth.func%C2%B7001

http://github.com/couchbaselabs/sync_gateway/channels.TimedSet.Copy

http://github.com/couchbaselabs/sync_gateway/channels

http://github.com/couchbaselabs/sync_gateway/channels.decodeChangeLog

http://github.com/gorilla/context.Set

http://github.com/gorilla/mux


Gzipped Responses

Sync Gateway clients are mobile, gzipped response makes a lot of sense.

Initial implementation, invoked compress/flate NewWriter on each response

Each flate.Writer consumed 1.4MB Rewritten to use sync.Pool

Acquire flate.Writer instances from pool Call Reset() and return to pool when done


Slices are Great… But…

Read 1MB of bytes off the wire into a buffer Parse the bytes, find the data you’re interested

in Return []byte with the key (say 16 bytes) The ENTIRE 1MB buffer is held in memory while

the slice is referenced. If these slices are long-lived and small relative to

their parent array, may be cheaper to copy them.

Go Routines / Channels


Too many channels?

Channels are synchronized

Synchronization has a cost

Use wisely, not haphazardly


Repeatable Builds


New/Prototype Projects

Bleeding Edge ‘go get’ everything Low friction Things break, and we fix them

But no one pretends this works for production!

24


Sync Gateway Builds

First Go project to need a solution to the problem Decided on a custom solution

Dependencies are vendored using git submodules Use wrapper scripts around native tools, go.sh, build.sh,

run.sh Scripts setup custom $GOPATH containing only

sync_gateway and its dependencies Works well for sync_gateway team

Separate $GOPATH can make for more complex integrations with IDE’s and tools like Go oracle


Projects integrating with Couchbase Server

Couchbase Server already has a well established process for repeatable builds using ‘repo’ All external dependencies are cloned into a separate

github organization “couchbasedeps” In our repo manifests, we get sources from couchbase

deps, but install them into $GOPATH using their canonical package name

<project name="protobuf" remote="couchbasedeps" revision="655cdfa588ea190e901bc5590e65d5621688847c" path="godeps/src/github.com/golang/protobuf"/>

http://github.com/golang/protobuf


Set $GOPATH and all Go tools are happy!


Using cgo


cgo on Windows Depends on gcc

No option to use MSVC We recently went to great pains to remove dependency

on GCC and now we had to add it back Considered dynamically loading DLL

High maintenance for wrappers when APIs change Current solution is to install gcc on Windows

build boxes


Reliable Core Dumps with crash inside cgo

With gdb attached, we see what we expect However, if no debugger is attached, resulting

core file does not have correct stack trace Using gccgo does work, but not comfortable shipping this

to customers

Thoughts…


Go at Couchbase, at a Crossroads



Real Problems, Real Solutions

Solve the problems Contribute back to the community

Solve the problems Don’t contribute them back

Don’t solve them Move away from Go in the future

The Couchbase Go SDKgocb


Couchbase SDKs

What does it mean to be a Couchbase SDK?

Cluster

Bucket

CRUDView

QueryN1QL Query

FunctionalManage connections to the bucket within the cluster for different services.Provide a core layer where IO can be managed and optimized.Provide a way to manage buckets.

APIinsertDesignDocument()flush()listDesignDocuments()

FunctionalHold on to cluster information such as topology.

APIReference Cluster ManagementopenBucket()info()disconnect()

FunctionalGive the application developer a concurrent API for basic (k-v) or document management

APIget()insert()upsert()remove()

FunctionalAllow for querying, execution of other directives such as defining indexes and checking on index state.

APIclient.NewN1QLQuery( “SELECT * FROM default LIMIT 5” ) .Consistency(gocouchbase.RequestPlus);

FunctionalAllow for view querying, building of queries and reasonable error handling from the cluster.

APIabucket.NewViewQuery().Limit().Stale()


Being True to Go Philosophy

Easy to read, write, understand with high-level concurrency (goroutines), type safety. Low level access if needed.

gocbcore

gocb

See more in a gist from Brett Lawson.

var user interface{}

bucket.Get("user_1", &user)fmt.Printf("Got user_1: %v\n", user)



signal := make(chan int)go func() {

var user interface{}bucket.Get("user_1",

&user)fmt.Printf("Got user_1:

%v\n", user)signal <- 1

}()go func() {




}()go func() {




}()for i := 0; i < 3; i++ {

<-signal}

// Note that this does not perform transcoding (ie: JSON marshalling)agent := bucket.IoRouter()signal := make(chan int)

_, err = agent.Get([]byte("user_1"), func(value []byte, flags uint32, cas uint64, err error) {

fmt.Printf("Got user_1: %v\n", string(value))

signal <- 1})



signal <- 1})



signal <- 1})

for i := 0; i < 3; i++ {<-signal

}

gocb is iterating toward interface stability

gocbcore will be a volatile interface for some time

https://gist.github.com/brett19/2db6395358f5eb4bfb23


JSON and Other Data Types

cbgo uses Go’s support for JSON internally

type BeerFull struct {Type string `json:"type"Ìd string

`json:"id,omitempty"`BreweryId string

`json:"brewery_id"`Name string `json:"name"`Description string

`json:"description"`Style string `json:"style"`Category string

`json:"category"Àbv float64 `json:"abv"Ìbu float64 `json:"ibu"`Srm float64 `json:"srm"Ùpc float64 `json:"upc"`

}

var beer BeerFull_, _, err := bucket.Get(id, &beer)if err != nil {

fmt.Fprintf(w, "Get Error: %v\n", err)return

}

// reference things like thisbeer.Name

Given this struct

You can unmarshal with a Get()


gocbcore – Where all of the Performance Tricks Live

High level concurrency in Go is great, but… Better still (for go) is avoiding the use of the

primitives

gocbcore is (mostly) lockless Has a concept of a “pipeline” and only two locks

within it Coarse grained “lock” as an RWMutex used only in the

case of death for cleanup Fine grain mutex on the operation list (memdOpMap) when

adding and removing items to the “pipeline”== minimal blocking of your goroutines

DemoTour of a Web Application


Current Status and Roadmap

Status Current version: 0.2.0 Release early and often to get user feedback, iterate

Roadmap Continue to iterate on 0.x releases adding features… N1QL Support Durability Requirements Complete test coverage && Fully Review Couchbase QE

tests Release 1.0!

Thanks!Marty Schoch – @mschoch

Matt Ingenthron – @ingenthr

Get Started with Couchbase Server 4.0: www.couchbase.com/beta

Get Trained on Couchbase: training.couchbase.com

Go’ing to the Next Level, Lessons from Go at Couchbase and Introducing the Go SDK: Couchbase...

Technology

Transcript of Go’ing to the Next Level, Lessons from Go at Couchbase and Introducing the Go SDK: Couchbase...