Developing High Performance Application with Aerospike & Go
-
Upload
chris-stivers -
Category
Software
-
view
654 -
download
3
Transcript of Developing High Performance Application with Aerospike & Go
![Page 1: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/1.jpg)
Developing High Performance
Applications using
![Page 2: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/2.jpg)
Aerospike Go
• Go Client Library
• CLI and Web Tools
• Benchmark Tools
• Hiring
![Page 3: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/3.jpg)
What is Aerospike?
Distributed Database
Key-Value Store
Secondary Indexes
User-Defined Functions
Large Data Types
![Page 4: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/4.jpg)
Why use Aerospike?
High Throughput
Low Latency
Highly Scalable
Highly Reliable
![Page 5: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/5.jpg)
Aerospike is High Throughput
0100000200000300000400000500000600000700000800000900000
10000001100000120000013000001400000150000016000001700000
Balanced Read-Heavy
Aerospike 3 (in-memory)
Aerospike 3 (persistent)
Aerospike 2
Cassandra
MongoDB
Couchbase 1.8
Couchbase 2.0
![Page 6: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/6.jpg)
Aerospike is Low Latency
0
2.5
5
7.5
10
0 50,000 100,000 150,000 200,000
Ave
rag
e L
ate
ncy,
ms
Throughput, ops/sec
Aerospike
![Page 7: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/7.jpg)
Who uses Aerospike?
theTradeDesk
![Page 8: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/8.jpg)
7 of the Top 20
Advertising
Platforms
Rank Name
1
2DoubleClick
2 Google AdSense
3 AppNexus
4 Google Remarketing
5 Yahoo Small Business
6 Google Publisher Tag
7 Openads/OpenX
8 Evidon
9Turn
10 Facebook Exchange
11 Rubicon Project
12 Pubmatic
13 The Trade Desk
14 BlueKai
15 Criteo
16 Casale Media
17 Rocket Fuel
18 Advertising.com (AOL)
19 ContextWeb
![Page 9: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/9.jpg)
Knows High Performance
![Page 10: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/10.jpg)
Performance Tips
Memory Allocation
Recycling
Pooling
![Page 11: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/11.jpg)
Angry GC
<http://blog.cloudflare.com/recycling-memory-buffers-in-go/>
Mellow GC
![Page 12: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/12.jpg)
Memory Allocation
• Avoid Dynamic AllocationNote: n resizes == (alloc + copy) * n
• Allocate exactly what you need upfront
• Its OK to allocate more than enough
• Garbage is your enemy
![Page 13: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/13.jpg)
Dynamic vs Static
var b bytes.Buffer
b.Write(a)
368 ns/op
b := bytes.NewBuffer(a)
11.5 ns/op
![Page 14: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/14.jpg)
Recycle
• Avoid GCNote: Releasing objects trigger GC
• Create and Reuse
• bytes.Buffer is slow for reuse
![Page 15: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/15.jpg)
Creating a Buffer and Copying Data
b := make([]byte, 256)
copy(b, a)
20.6 ns/op
b := bytes.NewBuffer(a)
11.5 ns/op
![Page 16: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/16.jpg)
Recycle
b:= make([]byte, 256)
copy(b, a)
10.6 ns/op
b := bytes.NewBuffer(a)
b.Reset()
b.Write(a)
25.8 ns/op
![Page 17: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/17.jpg)
Pooling
• Pools work great for short lived concurrent tasks.
• Size pools sufficiently
• Avoid sync.Pool for pooling objectsNote: Not ideal for long lives pool of objects. Cleans up with each GC
![Page 18: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/18.jpg)
sync.Pool
p := &sync.Pool{
New: NewBuffer,
}
b := p.Get().([]byte)
p.Put(b)
115 ns/op
![Page 19: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/19.jpg)
Channel-based Pool (FIFO)
type PoolChan struct {
buffers chan []byte
}
func (p *PoolChan) Get() []byte
func (p *PoolChan) Put(b []byte) bool
![Page 20: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/20.jpg)
Channel-based Pool
func (p *PoolChan) Get() []byte {
select {
case b := <-p.buffers:
return b
default:
}
return NewBuffer()
}
![Page 21: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/21.jpg)
Channel-based Pool
func (p *PoolChan) Put(b []byte) bool {
select {
case p.buffers <- b:
return true
default:
}
return false
}
![Page 22: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/22.jpg)
Channel-based Pool
p := &PoolChan{
buffers: make(chan []byte, 128),
}
b := p.Get()
p.Put(b)
67.4 ns/op
![Page 23: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/23.jpg)
Pooling
p := &sync.Pool{...}
b := p.Get().([]byte)
p.Put(b)
115 ns/op
p := &PoolChan{...}
b := p.Get()
p.Put(b)
67.4 ns/op
![Page 24: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/24.jpg)
Slice-based Pool (LIFO)
type PoolSlice struct {
buffers [][]byte
mutex sync.Mutex
}
func (p *PoolSlice) Get() []byte
func (p *PoolSlice) Put(b []byte) bool
![Page 25: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/25.jpg)
Slice-based Pool
func (p *PoolSlice) Get() []byte {
if len(p.buffers) == 0 {
return NewBuffer()
} else {
mutex.Lock()
b := p.buffers[len(p.buffers)-1]
p.buffers = p.buffers[0 : len(p.buffers)-1]
mutex.Unlock()
return b
}
}
![Page 26: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/26.jpg)
Slice-based Pool
func (p *PoolSlice) Put(b []byte) bool {
mutex.Lock()
p.buffers = append(p.buffers, b)
mutex.Unlock()
return true
}
![Page 27: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/27.jpg)
Slice-based Pool
p := &PoolSlice{
buffers: make([][]byte, 0, 128),
}
b := p.Get()
p.Put(b)
51.6 ns/op
![Page 28: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/28.jpg)
Pooling
p := &PoolSlice{...}
b := p.Get()
p.Put(b)
51.6 ns/op
p := &PoolChan{...}
b := p.Get()
p.Put(b)
67.4 ns/op
![Page 29: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/29.jpg)
Now what?
![Page 30: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/30.jpg)
Using Aerospike
Benchmark
Client Library
Data Model
API Highlights
![Page 31: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/31.jpg)
Client Library
github.com/aerospike/aerospike-client-go
![Page 32: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/32.jpg)
Data Model
namespace set key bins ttl gen
test demo x a: “abc”, b: 123
test demo y b: 345, c: [3,4,5]
test demo z a: “ghi”, c: [6,7,8]
... ... ... ...
Keys:
- Integer
- String
- Byte Array
Bin: Name -> Value
- Name:
- String
- Value:
- Integer (indexable)
- String (indexable)
- Byte Array
- List
- Map
- LDT (Map, List, Stack)
![Page 33: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/33.jpg)
Create a Client
// using default policy
client, err := NewClient("10.1.1.1", 3000)
// using custom policy
client, err := NewClientWithPolicy(policy, "10.1.1.1", 3000)
// using custom policy and host list
client, err := NewClientWithPolicy(policy,
NewHost("10.1.1.1", 3000), NewHost("10.1.1.2", 3001))
![Page 34: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/34.jpg)
Create a Key
// integer key
key, err := NewKey("test", "demo", 123)
// string key
key, err := NewKey("test", "demo","abc")
// byte array key
key, err := NewKey("test", "demo", []byte(uuid.NewRandom()))
![Page 35: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/35.jpg)
Write a Record using a Bin Array
bins := []Bin{
NewBin("name", "Bob"),
NewBin("age", 26),
NewBin("interests", []string{"golang"}),
NewBin("location", map[string]string{
"city": "New York",
"state": "New York",
}),
}
client.PutBins(policy, key, bins...)
![Page 36: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/36.jpg)
Write a Record using a BinMap
bins := BinMap{
"name": "Bob",
"age": 26,
"interests": []string{"golang"},
"location": map[string]string{
"city": "New York",
"state": "New York",
},
}
client.Put(policy, key, bins)
![Page 37: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/37.jpg)
Read a Record
rec, err := client.Get(policy, key)
println("Name:", rec.Bins["name"].(string))
![Page 38: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/38.jpg)
Query Records
stmt := NewStatement("test", "demo")
recordset, err := client.Query(policy, stmt)
for rec := range recordset.Records {
println("Name:", rec.Bins["name"].(string))
}
![Page 39: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/39.jpg)
Query Records with Filter
stmt := NewStatement("test", "demo")
stm.Addfilter(NewRangeFilter("age", 18, 34)
recordset, err := client.Query(policy, stmt)
for rec := range recordset.Records {
println("Name:", rec.Bins["name"].(string))
}
![Page 40: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/40.jpg)
Defining a User-Defined Function
// sample.lua
function store_if_larger(rec, bin, val)
if val > (rec[bin] or 0) then
r[bin] = b
aerospike:update(rec)
end
return r[bin]
end
![Page 41: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/41.jpg)
Calling a User-Defined Function
// stores x=1
err := client.PutBins(policy, key, NewBin("x", 1))
// stores x=2, returns 2
res, err := client.Execute(policy, key, "sample", "store_if_larger", NewValue("x"),
NewValue(2))
// returns 2
res, err := client.Execute(policy, key, "sample", "store_if_larger", NewValue("x"),
NewValue(1))
![Page 42: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/42.jpg)
Using a Large Stack (LDT)
// reference a large stack in database
lstack := NewLargeStack(client, policy, key, "timeline", "")
// push 1 million values on to the list
for i := 1; i <= 1000 * 1000; i++ {
err := lstack.Push(NewValue(i))
}
// pop the first 10
sub := lstack.Pop(10)
![Page 43: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/43.jpg)
Run the Benchmark
$ go install github.com/aerospike/aerospike-client-go/tools/benchmark
$ ./bin/benchmark
![Page 44: Developing High Performance Application with Aerospike & Go](https://reader033.fdocuments.in/reader033/viewer/2022042701/55a1506d1a28abd2488b4838/html5/thumbnails/44.jpg)
Learn More
• Aerospike <aerospike.com>
• Aerospike Projects <github.com/aerospike>
• Aerospike Go Client <github.com/aerospike/aerospike-client-go>
• Community Projects <aerospike.com/community/labs>
• Community Forums <discuss.aerospike.com>