Gophers Riding Elephants: Writing PostgreSQL tools in Go
-
Upload
aj-bahnken -
Category
Technology
-
view
176 -
download
0
Transcript of Gophers Riding Elephants: Writing PostgreSQL tools in Go
![Page 1: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/1.jpg)
Gophers Riding Elephants: Writing PostgreSQL Tools in Go
by AJ Bahnken, Senior Engineer @ Procore
![Page 2: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/2.jpg)
Who am I?Senior Engineer @ ProcoreWork on availability, performance, and (now mostly)securityBeen writing Go code actively for 2 years
Twitter:
Email: [email protected]
@ajvbahnken
![Page 3: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/3.jpg)
Who is this talk for?
![Page 4: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/4.jpg)
Overview of Go
![Page 5: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/5.jpg)
![Page 6: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/6.jpg)
Go was created by Google
![Page 7: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/7.jpg)
It was built for Google as well
![Page 8: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/8.jpg)
ReliableGood for teamsGood for building the kind of things they need to build
![Page 9: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/9.jpg)
"Less is exponentially more" by Rob Pike
Go is purposefully lacking certain features.
Link: https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html
![Page 10: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/10.jpg)
Statically TypedCompiledGarbage Collected
![Page 11: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/11.jpg)
package main
import "fmt"
func main() fmt.Println("Hello, 世界")
![Page 12: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/12.jpg)
Why Go with Postgres?
![Page 13: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/13.jpg)
1. Performance
![Page 14: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/14.jpg)
It's pretty fast
![Page 15: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/15.jpg)
Garbage collector is pretty solid
![Page 16: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/16.jpg)
Concurrency is a breezepackage main
import ( "fmt" "time" )
func say(s string) for i := 0; i < 5; i++ time.Sleep(100 * time.Millisecond) fmt.Println(s)
func main() go say("world") say("hello") // Allows for the goroutine to run, by blocking.
![Page 17: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/17.jpg)
$ go run main.go world hello hello world world hello hello world world hello
![Page 18: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/18.jpg)
2. Reliability
![Page 19: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/19.jpg)
Statically Typed (yay!)bool
string
int int8 int16 int32 int64 uint uint8 uint16 uint32 uint64 uintptr
byte // alias for uint8
rune // alias for int32 // represents a Unicode code point
float32 float64
complex64 complex128
![Page 20: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/20.jpg)
Simpletype EventProcessor struct eventQueue chan Event
func (ep *EventProcessor) Add(event Event) ep.eventQueue < event
func (ep *EventProcessor) Start() for event := <ep.eventQueue go event.Process()
![Page 21: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/21.jpg)
Testing is simple and built in + race detector$ ls processing.go processing_test.go utils.go
$ go test PASS ok ~/pgnetdetective/processing 0.165s
$ go test race PASS ok ~/pgnetdetective/processing 2.133s
![Page 22: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/22.jpg)
Error handling instead of exceptionsfunc MyFunc() (string, error) str, err := run() if err != nil return "", err return str, nil
func MustMyFunc() string str, err := run() if err != nil panic("run() returned an err: "+err.String()) return str
![Page 23: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/23.jpg)
3. Ease of Use
![Page 24: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/24.jpg)
Tooling
(Gofmt, testing, godocs, go build/run, vim-go)
![Page 25: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/25.jpg)
Familiarity
![Page 26: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/26.jpg)
Library support and ease of installation$ go get github.com/urfave/cli
![Page 27: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/27.jpg)
Distribute a single binary anywhere$ go build $ file dbduke dbduke: MachO 64bit executable x86_64
$ GOOS=linux go build $ file dbduke dbduke: ELF 64bit LSB executable, x8664, version 1 (SYSV), \ statically linked, not stripped
$ GOOS=linux GOARCH=386 go build $ file dbduke dbduke: ELF 32bit LSB executable, Intel 80386, version 1 \ (SYSV), statically linked, not stripped
![Page 28: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/28.jpg)
PerformanceReliability
Ease of Use
![Page 29: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/29.jpg)
Interacting withPostgres in Go
![Page 31: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/31.jpg)
Provides core interface for interacting with SQL databases
Open / CloseBegin / Rollback / CommitExec / Query / QueryRowPing / Connection Pooling
![Page 32: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/32.jpg)
go get github.com/lib/pq
![Page 33: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/33.jpg)
package main
import ( "database/sql" "fmt" _ "github.com/lib/pq" )
func main() dbUrl := "postgres://postgres@localhost:5432/postgres" db, err := sql.Open("postgres", dbUrl) if err != nil panic(err.String())
var result int err = db.QueryRow('SELECT 1').Scan(&result) if err != nil panic(err.String())
fmt.Printf("1 == %d", result)
![Page 35: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/35.jpg)
Example #1pgnetdetective
https://github.com/procore/pgnetdetective
![Page 36: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/36.jpg)
![Page 37: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/37.jpg)
![Page 38: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/38.jpg)
![Page 39: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/39.jpg)
![Page 40: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/40.jpg)
?????
![Page 41: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/41.jpg)
![Page 42: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/42.jpg)
![Page 43: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/43.jpg)
tcpdump n w ~/pg.cap i any port 5432
~1GB every 10 seconds
![Page 44: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/44.jpg)
We needed something faster,
so I decided to rewrite it in Go
![Page 45: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/45.jpg)
https://github.com/google/gopacket
Provides packet processing capabilities for Go
![Page 46: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/46.jpg)
// If the destination port is 5432... if tcp.DstPort == 5432 // And the packet payload starts with P or Q... raw = fmt.Sprintf("%s", tcp.Payload) if strings.HasPrefix(raw, "P") || strings.HasPrefix(raw, "Q") // It is a Parse or Query packet, therefore it contains a Query combinedQueryMetrics.Add( metrics.New( NormalizeQuery(raw), 1, ip.SrcIP, tcp.Seq, ), ) else if tcp.SrcPort == 5432 && tcp.ACK responses = append(responses, &ResponsePacket DstIP: ip.DstIP, Ack: tcp.Ack, Size: uint64(len(tcp.Payload)), )
![Page 47: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/47.jpg)
So I got some output like this:******* Query ******* Query: SELECT attr.attname FROM pg_attribute attr INNER JOIN pg_constraint cons ON attr.attrelid = cons.conrelid AND attr.attnum = any(cons.conkey) WHERE cons.contype = p AND cons.conrelid = "drawing_log_imports"::regclass TotalNetBytes: 170 MB TotalResponsePackets: 64041 TotalQueryPackets: 63
![Page 48: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/48.jpg)
ummm, catalog queries?
![Page 49: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/49.jpg)
Introducing: Resquehttps://github.com/resque/resque
http://resque.github.io/
![Page 50: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/50.jpg)
~10,000 jobs per hour1-8 tables being touched per jobaverage of 20 columns per table.
![Page 51: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/51.jpg)
During spikes, this can get up to 120MB per second.
On to Sidekiq we go...
![Page 52: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/52.jpg)
Where Go won with pgnetdetective:Performance
Community (Ease of Use)
![Page 53: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/53.jpg)
![Page 54: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/54.jpg)
Example #2 dbduke(not open source + still under active development)
![Page 55: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/55.jpg)
Context:
1. We restore staging/qa/testing databases frequently2. It's important that they successfully restore
![Page 56: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/56.jpg)
Problem:
1. When restores fail, productivity dies2. The process of kicking restores off by hand is faulty
![Page 57: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/57.jpg)
Further Context for a Solution:
1. Restores sometimes fail from easily recoverable errors
![Page 58: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/58.jpg)
A tool for making restores of Postgresdatabases manageable and fault tolerant.
![Page 59: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/59.jpg)
A tool for making restores of Postgresdatabases manageable and fault tolerant.
![Page 60: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/60.jpg)
ManageableRun dbduke as a daemon with jobs
![Page 61: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/61.jpg)
$ dbduke jobs DBDuke * restore 35e1ca93936b4c738812b1a69d708791 database: postgres dumpfile: /data/litedump.dmp started: 17:19:59 Tue Oct 11, 2016 0700 flags: nobigtables maintenance
![Page 62: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/62.jpg)
A tool for making restores of Postgresdatabases manageable and fault tolerant.
![Page 63: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/63.jpg)
Fault ToleranceTreat restores as a state machine
and recover from failure states
![Page 64: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/64.jpg)
![Page 65: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/65.jpg)
Error handling in practice:
1. Error out2. Log a warning3. Retry with timeout (with or without backoff)
![Page 66: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/66.jpg)
Error outdb, err := sql.Open("postgres", dbUrl) if err != nil log.Fatalf("Could not open postgres db @ `%s`", dbUrl)
![Page 67: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/67.jpg)
Log warningquery := "DROP SCHEMA IF EXISTS _migration CASCADE" _, err = db.Exec(query) if err != nil log.Warnf("Query `%s` failed with err: %v", query, err)
![Page 68: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/68.jpg)
Retry with timeout (without backoff)func (r *Restorer) BlockTillNotInUse() if r.State == state.InUse log.Warn("State is currently InUse. Going into retry loop.") for time.Sleep(time.Second * 15)
r.QuitIfTimeout()
currentState, err := state.GetCurrentState() if err != nil log.Errorf( "Error getting current state. Err: %v", err, ) break
if currentState != state.InUse r.State = currentState break
![Page 69: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/69.jpg)
Manageability + Fault Tolerance
Go makes it easy! ™
![Page 70: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/70.jpg)
Where Go won with dbduke:Error Handling (Reliability)
Concurrency (Performance/Ease of Use)
![Page 71: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/71.jpg)
In ConclusionIn the context of tool building
Go = Reliability, Performance, and Ease ofUse
![Page 72: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/72.jpg)
![Page 73: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/73.jpg)
![Page 74: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/74.jpg)
![Page 77: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/77.jpg)
Further Resources
Tour of GoEffective Go (required reading)Great intoduction to using SQL within GoWhy we import drivers with '_'
![Page 78: Gophers Riding Elephants: Writing PostgreSQL tools in Go](https://reader036.fdocuments.in/reader036/viewer/2022062400/58ce91851a28ab8c3b8b58d7/html5/thumbnails/78.jpg)
Sources
Performance Graph