DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group...
Transcript of DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group...
![Page 1: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/1.jpg)
HotCloud '17 | 2017-07-10 | Page 1
DAL: A Locality-Optimizing Distributed Shared Memory System
Gábor Németh, Dániel Géhbeger and Péter Mátray
Ericsson Research
{gabor.a.nemeth, daniel.gehberger, peter.matray}@ericsson.com
![Page 2: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/2.jpg)
HotCloud '17 | 2017-07-10 | Page 2
› Critical applications moving to the cloud
– Telecom applications
– Industrial IoT, cloud controlled collaborative systems
› Tight end-to-end latency requirements
› Handling continuous influx of data
– Heavy state sharing and cross communication
– Various data access patterns
Introduction
![Page 3: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/3.jpg)
HotCloud '17 | 2017-07-10 | Page 3
For a fast data store, transport costs dominate response times
Handle the locality of each data item separately & seamlessly move
data to the worker process accessing it
Flexible state externalization for critical applications
Idea behind DAL
![Page 4: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/4.jpg)
HotCloud '17 | 2017-07-10 | Page 4
– Local data access
› Lock-free shared memory IPC
› Direct memory read
– Remote data access
› Local server acts as a proxy
› DPDK is used to bypass the
kernel
› UDP based transport
High-level architecture
![Page 5: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/5.jpg)
HotCloud '17 | 2017-07-10 | Page 5
Locality optimized Key-value access
› Single key sharding
› Two-phase lookup: key location value
› Location is cached
Client Value
Get location 1
Key to location service
Access3
DAL lib2
Cache the location
› Distributed key space
› Server roles: key/data/combined
› Value can be moved separately of key location
› Automatic move to dominant accessor
![Page 6: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/6.jpg)
HotCloud '17 | 2017-07-10 | Page 6
Data access patterns
› Key-value
› Messaging
– PUB-SUB
› Watch for changes of a value
Application B
Application C
Value
Write1Notify2
Application AApplication AApplication A
![Page 7: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/7.jpg)
HotCloud '17 | 2017-07-10 | Page 7
Data access patterns
› Key-value
› Messaging
– PUB-SUB
– Group PUB-SUB
› Watch for changes of a value
› Multiple parallel groups are all notified
› Load sharing in groups
- Round-robin
Value
Write1Notify2
Application AApplication AApplication A
Application AApplication AApplication C
Application AApplication AApplication B
![Page 8: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/8.jpg)
HotCloud '17 | 2017-07-10 | Page 8
Data access patterns
› Key-value
› Messaging
– PUB-SUB
– Group PUB-SUB
› Watch for changes of a value
› Multiple parallel groups are all notified
› Load sharing in groups
- Round-robin
- StickyValue
Write1
Notify2
Application AApplication AApplication A
Application B
Application B
Application B
Application C
Application C
![Page 9: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/9.jpg)
HotCloud '17 | 2017-07-10 | Page 9
Data access patterns
› Key-value
› Messaging
– PUB-SUB
– Group PUB-SUB
› Request-response communication
– Combined with load sharing
Request
Notify on write
ResponseResponse
Application Service
1
34
ServiceServiceRequest
2
Notify on write
![Page 10: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/10.jpg)
HotCloud '17 | 2017-07-10 | Page 10
Data access patterns
› Key-value
› Messaging
– PUB-SUB
– Group PUB-SUB
› Request-response communication
› Decoupled Producers & Consumers
Application BValue
Key-valueApplication C
Application A
WriteMessaging
![Page 11: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/11.jpg)
HotCloud '17 | 2017-07-10 | Page 11
› Random access of 1 million keys with 100 byte values
› Throughput
› 1.6M local write
› 0.9M remote read or write
› RAMCloud
› Open source key-value store
› Comparable remote access
› No local access
Evaluation
Comparison Local [µs] Remote [µs]
RAMCloud using InfiniBand [3] - 4.7
RAMCloud in our lab - 21.15
Redis 18 52
[3] Ousterhout et.al.: The RAMCloud Storage System (ACM TOCS 2015 33(3), 7)
Elementary data operations 50% [µs] 99% [µs]
Local read 0.7 1.1
Local write 1.1 1.7
Remote read or write 21.9 22.8
![Page 12: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/12.jpg)
HotCloud '17 | 2017-07-10 | Page 12
Stateless applicationSingle session
Host 1
p1blue
emulatorp1, hash(p1)
p1 ++
Host 2
› Application with geographic partitioning
› Sticky DAL messaging
› State externalization
![Page 13: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/13.jpg)
HotCloud '17 | 2017-07-10 | Page 13
Host 2Host 1
blue
emulator
green++
p1
p1
› Application with geographic partitioning
› Sticky DAL messaging
› State externalization
Stateless applicationSingle session
Host 1
![Page 14: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/14.jpg)
HotCloud '17 | 2017-07-10 | Page 14
Host 2Host 1
p1blue
emulator
green
p1 ++
› Application with geographic partitioning
› Sticky DAL messaging
› State externalization
› Seamless state handover
Stateless applicationSingle session
Host 1
![Page 15: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/15.jpg)
HotCloud '17 | 2017-07-10 | Page 15
Stateless applicationSingle session
Host 2Host 1
p1blue
emulator
green
p1 ++
Host 1
![Page 16: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/16.jpg)
HotCloud '17 | 2017-07-10 | Page 16
Stateless applicationMass evaluation
Startup Stable Scaling Stable
![Page 17: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/17.jpg)
HotCloud '17 | 2017-07-10 | Page 17
› DAL achieves 1 µs data access
– Per key automatic locality optimization
– Shared memory communication
› State externalization for latency sensitive applications
– Telecom session handling, industrial IoT
› Discussion / current focus areas
– Locations have to be cached: key-space must be partitionable
– Asynchronous replication
– More advanced optimization algorithms
– Data & key server scaling
– More use cases…?
Conclusions
![Page 18: DAL: A Locality-Optimizing Distributed Shared Memory System · ›Messaging –PUB-SUB –Group PUB-SUB ›Watch for changes of a value ›Multiple parallel groups are all notified](https://reader034.fdocuments.in/reader034/viewer/2022042710/5f5b1f7339bd483ab872ac6e/html5/thumbnails/18.jpg)
HotCloud '17 | 2017-07-10 | Page 18