Why does Microsoft care about NoSQL, SQL and Polyglot Persistence?
Developing polyglot persistence applications (svcc, svcc2013)
-
Upload
chris-richardson -
Category
Technology
-
view
788 -
download
2
description
Transcript of Developing polyglot persistence applications (svcc, svcc2013)
![Page 1: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/1.jpg)
DEVELOPING POLYGLOT PERSISTENCE APPLICATIONS
Chris RichardsonAuthor of POJOs in ActionFounder of the original CloudFoundry.com @crichardson [email protected]://plainoldobjects.com
![Page 2: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/2.jpg)
@crichardson
Presentation goalThe benefits and drawbacks of
polyglot persistence and
How to design applications that use this approach
![Page 3: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/3.jpg)
@crichardson
About Chris
![Page 4: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/4.jpg)
@crichardson
(About Chris)
![Page 5: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/5.jpg)
@crichardson
About Chris()
![Page 6: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/6.jpg)
@crichardson
About Chris
![Page 7: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/7.jpg)
@crichardson
About Chris
http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
![Page 8: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/8.jpg)
@crichardson
About Chris
![Page 9: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/9.jpg)
@crichardson
Agenda
• Why polyglot persistence?
• Using Redis as a cache
• Optimizing queries using Redis materialized views
• Synchronizing MySQL and Redis
• Tracking changes to entities
![Page 10: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/10.jpg)
@crichardson
Food to Go
• Take-out food delivery service
• “Launched” in 2006
![Page 11: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/11.jpg)
@crichardson
Food To Go Architecture
Order taking
Restaurant Management
MySQL Database
CONSUMER RESTAURANT OWNER
![Page 12: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/12.jpg)
@crichardson
Success ⇒ Growth challenges
• Increasing traffic
• Increasing data volume
• Distribute across a few data centers
• Increasing domain model complexity
![Page 13: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/13.jpg)
@crichardson
Limitations of relational databases
• Scalability
• Distribution
• Schema updates
• O/R impedance mismatch
• Handling semi-structured data
![Page 14: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/14.jpg)
@crichardson
Solution: Spend $$$ - high-end databases and servers
http://www.powerandmotoryacht.com/megayachts/megayacht-musashi
![Page 15: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/15.jpg)
@crichardson
Solution: Spend $$$ - open-source stack + DevOps people
http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
![Page 16: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/16.jpg)
@crichardson
Solution: Use NoSQL
Benefits
• Higher performance
• Higher scalability
• Richer data-model
• Schema-less
Drawbacks
• Limited transactions
• Limited querying
• Relaxed consistency
• Unconstrained data
![Page 17: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/17.jpg)
@crichardson
Example NoSQL DatabasesDatabase Key features
Cassandra Extensible column store, very scalable, distributed
MongoDB Document-oriented, fast, scalable
Redis Key-value store, very fast
http://nosql-database.org/ lists 122+ NoSQL databases
![Page 18: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/18.jpg)
@crichardson
Redis: Fast and Scalable
Redis master
Redis slave Redis slave
Filesystem
ClientIn-memory
= FAST
Durable
Redis master
Redis slave Redis slave
Filesystem
Consistent Hashing
![Page 19: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/19.jpg)
@crichardson
Redis: advanced key/value store
K1 V1
K2 V2
... ...
String SET, GET
List LPUSH, LPOP, ...
Set SADD, SMEMBERS, ..
Sorted Set
Hashes
ZADD, ZRANGE
HSET, HGET
![Page 20: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/20.jpg)
@crichardson
Sorted sets
myset a
5.0
b
10.
Key Value
ScoreMembers are sorted by score
![Page 21: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/21.jpg)
@crichardson
Adding members to a sorted setRedis Server
zadd myset 5.0 a myset a
5.0
Key Score Value
![Page 22: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/22.jpg)
@crichardson
Adding members to a sorted setRedis Server
zadd myset 10.0 b myset a
5.0
b
10.
![Page 23: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/23.jpg)
@crichardson
Adding members to a sorted setRedis Server
zadd myset 1.0 c myset a
5.0
b
10.
c
1.0
![Page 24: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/24.jpg)
@crichardson
Retrieving members by index range
Redis Server
zrange myset 0 1
myset a
5.0
b
10.
c
1.0ac
Key StartIndex
EndIndex
![Page 25: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/25.jpg)
@crichardson
Retrieving members by score
Redis Server
zrangebyscore myset 1 6
myset a
5.0
b
10.
c
1.0ac
Key Min value
Max value
![Page 26: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/26.jpg)
@crichardson
Redis use cases• Replacement for Memcached
• Session state
• Cache of data retrieved from system of record (SOR)
• Replica of SOR for queries needing high-performance
• Handling tasks that overload an RDBMS
• Hit counts - INCR
• Most recent N items - LPUSH and LTRIM
• Randomly selecting an item – SRANDMEMBER
• Queuing – Lists with LPOP, RPUSH, ….
• High score tables – Sorted sets and ZINCRBY
• …
![Page 27: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/27.jpg)
@crichardson
The future is polyglot
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
e.g. Netflix• RDBMS• SimpleDB• Cassandra• Hadoop/Hbase
![Page 28: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/28.jpg)
@crichardson
Benefits
RDBMS
ACID transactionsRich queries...
+
NoSQL
PerformanceScalability...
![Page 29: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/29.jpg)
@crichardson
Drawbacks
Complexity
DevelopmentOperational
![Page 30: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/30.jpg)
@crichardson
Polyglot Pattern: Cache
Order taking
Restaurant Management
MySQL Database
CONSUMER RESTAURANT OWNER
RedisCache
First
Second
![Page 31: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/31.jpg)
@crichardson
Redis: Timeline
Polyglot Pattern: write to SQL and NoSQL
Follower 1
Follower 2
Follower 3 ...
LPUSHXLTRIM
MySQL
TweetsFollowers
INSERT
NEW TWEET
Follows
Avoids expensive
MySQL joins
![Page 32: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/32.jpg)
@crichardson
Polyglot Pattern: replicate from MySQL to NoSQL
QueryService
Update Service
MySQL Database
Reader Writer
Redis
Systemof RecordMaterialized
view query() update()replicate
anddenormalize
![Page 33: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/33.jpg)
@crichardson
Agenda
• Why polyglot persistence?
• Using Redis as a cache
• Optimizing queries using Redis materialized views
• Synchronizing MySQL and Redis
• Tracking changes to entities
![Page 34: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/34.jpg)
@crichardson
Food to Go – Domain model (partial)class Restaurant { long id; String name; Set<String> serviceArea; Set<TimeRange> openingHours; List<MenuItem> menuItems;}
class MenuItem { String name; double price;}
class TimeRange { long id; int dayOfWeek; int openTime; int closeTime;}
![Page 35: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/35.jpg)
@crichardson
Database schemaID Name …
1 Ajanta
2 Montclair Eggshop
Restaurant_id zipcode
1 94707
1 94619
2 94611
2 94619
Restaurant_id dayOfWeek openTime closeTime
1 Monday 1130 1430
1 Monday 1730 2130
2 Tuesday 1130 …
RESTAURANT table
RESTAURANT_ZIPCODE table
RESTAURANT_TIME_RANGE table
![Page 36: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/36.jpg)
@crichardson
RestaurantRepository
public interface RestaurantRepository { void addRestaurant(Restaurant restaurant); Restaurant findById(long id); ...}
Food To Go will have scaling eventually issues
![Page 37: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/37.jpg)
@crichardson
Increase scalability by caching
Order taking
Restaurant Management
MySQL Database
CONSUMER RESTAURANT OWNER
Cache
![Page 38: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/38.jpg)
@crichardson
Caching Options
• Where:
• Hibernate 2nd level cache
• Explicit calls from application code
• Caching aspect
• Cache technologies: Ehcache, Memcached, Infinispan, ...
Redis is also an option
![Page 39: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/39.jpg)
@crichardson
Using Redis as a cache
• Spring 3.1 cache abstraction
• Annotations specify which methods to cache
• CacheManager - pluggable back-end cache
• Spring Data for Redis
• Simplifies the development of Redis applications
• Provides RedisTemplate (analogous to JdbcTemplate)
• Provides RedisCacheManager
![Page 40: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/40.jpg)
@crichardson
Using Spring 3.1 Caching@Servicepublic class RestaurantManagementServiceImpl implements RestaurantManagementService {
private final RestaurantRepository restaurantRepository;
@Autowired public RestaurantManagementServiceImpl(RestaurantRepository restaurantRepository) { this.restaurantRepository = restaurantRepository; } @Override public void add(Restaurant restaurant) { restaurantRepository.add(restaurant); }
@Override @Cacheable(value = "Restaurant") public Restaurant findById(int id) { return restaurantRepository.findRestaurant(id); }
@Override @CacheEvict(value = "Restaurant", key="#restaurant.id") public void update(Restaurant restaurant) { restaurantRepository.update(restaurant); }
Cache result
Evict from cache
![Page 41: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/41.jpg)
@crichardson
Configuring the Redis Cache Manager
<cache:annotation-driven />
<bean id="cacheManager" class="org.springframework.data.redis.cache.RedisCacheManager" > <constructor-arg ref="restaurantTemplate"/> </bean>
Enables caching
Specifies CacheManager implementation
The RedisTemplate used to access Redis
![Page 42: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/42.jpg)
@crichardson
TimeRange MenuItem
Domain object to key-value mapping?
Restaurant
TimeRange MenuItem
ServiceArea
K1 V1
K2 V2
... ...
![Page 43: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/43.jpg)
@crichardson
RedisTemplate handles the mapping
• Principal API provided by Spring Data to Redis
• Analogous to JdbcTemplate
• Encapsulates boilerplate code, e.g. connection management
• Maps Java objects ⇔ Redis byte[]’s
![Page 44: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/44.jpg)
@crichardson
Serializers: object ⇔ byte[]
• RedisTemplate has multiple serializers
• DefaultSerializer - defaults to JdkSerializationRedisSerializer
• KeySerializer
• ValueSerializer
• ...
![Page 45: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/45.jpg)
@crichardson
Serializing a Restaurant as JSON@Configurationpublic class RestaurantManagementRedisConfiguration {
@Autowired private RestaurantObjectMapperFactory restaurantObjectMapperFactory; private JacksonJsonRedisSerializer<Restaurant> makeRestaurantJsonSerializer() { JacksonJsonRedisSerializer<Restaurant> serializer =
new JacksonJsonRedisSerializer<Restaurant>(Restaurant.class); ... return serializer; }
@Bean @Qualifier("Restaurant") public RedisTemplate<String, Restaurant> restaurantTemplate(RedisConnectionFactory factory) { RedisTemplate<String, Restaurant> template = new RedisTemplate<String, Restaurant>(); template.setConnectionFactory(factory); JacksonJsonRedisSerializer<Restaurant> jsonSerializer = makeRestaurantJsonSerializer(); template.setValueSerializer(jsonSerializer); return template; }
}
Serialize restaurants using Jackson JSON
![Page 46: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/46.jpg)
@crichardson
Caching with Redis
Order taking
Restaurant Management
MySQL Database
CONSUMER RESTAURANT OWNER
RedisCacheFirst Second
![Page 47: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/47.jpg)
@crichardson
Storing restaurants in Redis
...JSON... restaurant:1
![Page 48: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/48.jpg)
@crichardson
Agenda
• Why polyglot persistence?
• Using Redis as a cache
• Optimizing queries using Redis materialized views
• Synchronizing MySQL and Redis
• Tracking changes to entities
![Page 49: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/49.jpg)
@crichardson
Finding available restaurantsAvailable restaurants =
Serve the zip code of the delivery addressAND
Are open at the delivery time
public interface AvailableRestaurantRepository {
List<AvailableRestaurant> findAvailableRestaurants(Address deliveryAddress, Date deliveryTime); ...}
![Page 50: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/50.jpg)
@crichardson
Finding available restaurants on Monday, 6.15pm for 94619 zipcode
Straightforward three-way join
select r.*from restaurant r inner join restaurant_time_range tr on r.id =tr.restaurant_id inner join restaurant_zipcode sa on r.id = sa.restaurant_id where ’94619’ = sa.zip_code and tr.day_of_week=’monday’ and tr.openingtime <= 1815 and 1815 <= tr.closingtime
![Page 51: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/51.jpg)
@crichardson
How to scale queries?
![Page 52: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/52.jpg)
@crichardson
Option #1: Query caching
• [ZipCode, DeliveryTime] ⇨ list of available restaurants
BUT
• Long tail queries
• Update restaurant ⇨ Flush entire cache
Ineffective
![Page 53: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/53.jpg)
@crichardson
Option #2: Master/Slave replication
MySQL Master
MySQL Slave 1
MySQL Slave 2
MySQL Slave N
Writes Consistent reads
Queries(Inconsistent reads)
![Page 54: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/54.jpg)
@crichardson
Master/Slave replication
• Mostly straightforward
BUT
• Assumes that SQL query is efficient
• Complexity of administration of slaves
• Doesn’t scale writes
![Page 55: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/55.jpg)
@crichardson
Option #3: Redis materialized views
Order taking
Restaurant Management
MySQL Database
CONSUMER RESTAURANT OWNER
CacheRedis
Systemof RecordCopy
findAvailable()update()
![Page 56: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/56.jpg)
@crichardson
BUT how to implement findAvailableRestaurants() with Redis?!
?select r.*from restaurant r inner join restaurant_time_range tr on r.id =tr.restaurant_id inner join restaurant_zipcode sa on r.id = sa.restaurant_id where ’94619’ = sa.zip_code and tr.day_of_week=’monday’ and tr.openingtime <= 1815 and 1815 <= tr.closingtime
K1 V1
K2 V2
... ...
![Page 57: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/57.jpg)
@crichardson
Solution: Build an index using sorted sets and ZRANGEBYSCORE
ZRANGEBYSCORE myset 1 6
select valuefrom sorted_setwhere key = ‘myset’ and score >= 1 and score <= 6
=
key value score
sorted_set
![Page 58: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/58.jpg)
@crichardson
select r.*from restaurant r inner join restaurant_time_range tr on r.id =tr.restaurant_id inner join restaurant_zipcode sa on r.id = sa.restaurant_id where ’94619’ = sa.zip_code and tr.day_of_week=’monday’ and tr.openingtime <= 1815 and 1815 <= tr.closingtime
select valuefrom sorted_setwhere key = ? and score >= ? and score <= ?
?
How to transform the SELECT statement?
![Page 59: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/59.jpg)
@crichardson
We need to denormalize
Think materialized view
![Page 60: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/60.jpg)
@crichardson
Simplification #1: Denormalization
Restaurant_id Day_of_week Open_time Close_time Zip_code
1 Monday 1130 1430 947071 Monday 1130 1430 946191 Monday 1730 2130 947071 Monday 1730 2130 946192 Monday 0700 1430 94619…
SELECT restaurant_id FROM time_range_zip_code WHERE day_of_week = ‘Monday’ AND zip_code = 94619 AND 1815 < close_time AND open_time < 1815
Simpler query:§ No joins§ Two = and two <
![Page 61: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/61.jpg)
@crichardson
Simplification #2: Application filtering
SELECT restaurant_id, open_time FROM time_range_zip_code WHERE day_of_week = ‘Monday’ AND zip_code = 94619 AND 1815 < close_time AND open_time < 1815
Even simpler query• No joins• Two = and one <
![Page 62: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/62.jpg)
@crichardson
Simplification #3: Eliminate multiple =’s with concatenation
SELECT restaurant_id, open_time FROM time_range_zip_code WHERE zip_code_day_of_week = ‘94619:Monday’ AND 1815 < close_time
Restaurant_id Zip_dow Open_time Close_time
1 94707:Monday 1130 14301 94619:Monday 1130 14301 94707:Monday 1730 21301 94619:Monday 1730 21302 94619:Monday 0700 1430…
key
range
![Page 63: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/63.jpg)
@crichardson
Simplification #4: Eliminate multiple RETURN VALUES with concatenation
SELECT open_time_restaurant_id, FROM time_range_zip_code WHERE zip_code_day_of_week = ‘94619:Monday’ AND 1815 < close_time
zip_dow open_time_restaurant_id close_time
94707:Monday 1130_1 1430
94619:Monday 1130_1 1430
94707:Monday 1730_1 2130
94619:Monday 1730_1 2130
94619:Monday 0700_2 1430
...
✔
![Page 64: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/64.jpg)
@crichardson
zip_dow open_time_restaurant_id close_time
94707:Monday 1130_1 1430
94619:Monday 1130_1 1430
94707:Monday 1730_1 2130
94619:Monday 1730_1 2130
94619:Monday 0700_2 1430
...
Sorted Set [ Entry:Score, …]
[1130_1:1430, 1730_1:2130]
[0700_2:1430, 1130_1:1430, 1730_1:2130]
Using a Redis sorted set as an index
Key
94619:Monday
94707:Monday
94619:Monday 0700_2 1430
![Page 65: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/65.jpg)
@crichardson
Querying with ZRANGEBYSCORE
ZRANGEBYSCORE 94619:Monday 1815 2359è
{1730_1}
1730 is before 1815 è Ajanta is open
Delivery timeDelivery zip and day
Sorted Set [ Entry:Score, …]
[1130_1:1430, 1730_1:2130]
[0700_2:1430, 1130_1:1430, 1730_1:2130]
Key
94619:Monday
94707:Monday
![Page 66: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/66.jpg)
@crichardson
Adding a Restaurant
Text
@Componentpublic class AvailableRestaurantRepositoryImpl implements AvailableRestaurantRepository {
@Override public void add(Restaurant restaurant) { addRestaurantDetails(restaurant); addAvailabilityIndexEntries(restaurant); }
private void addRestaurantDetails(Restaurant restaurant) { restaurantTemplate.opsForValue().set(keyFormatter.key(restaurant.getId()), restaurant); } private void addAvailabilityIndexEntries(Restaurant restaurant) { for (TimeRange tr : restaurant.getOpeningHours()) { String indexValue = formatTrId(restaurant, tr); int dayOfWeek = tr.getDayOfWeek(); int closingTime = tr.getClosingTime(); for (String zipCode : restaurant.getServiceArea()) { redisTemplate.opsForZSet().add(closingTimesKey(zipCode, dayOfWeek), indexValue,
closingTime); } } }
key member
score
Store as JSON
![Page 67: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/67.jpg)
@crichardson
Finding available Restaurants@Componentpublic class AvailableRestaurantRepositoryImpl implements AvailableRestaurantRepository { @Override public List<AvailableRestaurant>
findAvailableRestaurants(Address deliveryAddress, Date deliveryTime) { String zipCode = deliveryAddress.getZip(); int dayOfWeek = DateTimeUtil.dayOfWeek(deliveryTime); int timeOfDay = DateTimeUtil.timeOfDay(deliveryTime); String closingTimesKey = closingTimesKey(zipCode, dayOfWeek);
Set<String> trsClosingAfter = redisTemplate.opsForZSet().rangeByScore(closingTimesKey, timeOfDay, 2359);
Set<String> restaurantIds = new HashSet<String>(); for (String tr : trsClosingAfter) { String[] values = tr.split("_"); if (Integer.parseInt(values[0]) <= timeOfDay) restaurantIds.add(values[1]); } Collection<String> keys = keyFormatter.keys(restaurantIds); return availableRestaurantTemplate.opsForValue().multiGet(keys); }
Find those that close after
Filter out those that open after
Retrieve open restaurants
![Page 68: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/68.jpg)
@crichardson
Representing restaurants in Redis
...JSON... restaurant:1
94619:Monday [0700_2:1430, 1130_1:1430, ...]
94619:Tuesday [0700_2:1430, 1130_1:1430, ...]
![Page 69: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/69.jpg)
@crichardson
NoSQL ⇒ Denormalized representation for each query
![Page 70: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/70.jpg)
@crichardson
Sorry Ted!
http://en.wikipedia.org/wiki/Edgar_F._Codd
![Page 71: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/71.jpg)
@crichardson
Agenda
• Why polyglot persistence?
• Using Redis as a cache
• Optimizing queries using Redis materialized views
• Synchronizing MySQL and Redis
• Tracking changes to entities
![Page 72: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/72.jpg)
@crichardson
MySQL & Redis need to be consistent
![Page 73: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/73.jpg)
@crichardson
Two-Phase commit is not an option
• Redis does not support it
• Even if it did, 2PC is best avoided http://www.infoq.com/articles/ebay-scalability-best-practices
![Page 74: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/74.jpg)
@crichardson
AtomicConsistentIsolatedDurable
Basically AvailableSoft stateEventually consistent
BASE: An Acid Alternative http://queue.acm.org/detail.cfm?id=1394128
![Page 75: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/75.jpg)
@crichardson
Updating Redis #FAILbegin MySQL transaction update MySQL update Redisrollback MySQL transaction
Redis has updateMySQL does not
begin MySQL transaction update MySQLcommit MySQL transaction<<system crashes>> update Redis
MySQL has updateRedis does not
![Page 76: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/76.jpg)
@crichardson
Updating Redis reliablyStep 1 of 2
begin MySQL transactionupdate MySQLqueue CRUD event in MySQL
commit transaction
ACID
Event IdOperation: Create, Update, DeleteNew entity state, e.g. JSON
![Page 77: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/77.jpg)
@crichardson
Updating Redis reliably Step 2 of 2
for each CRUD event in MySQL queue get next CRUD event from MySQL queue
If CRUD event is not duplicate then Update Redis (incl. eventId)end ifbegin MySQL transaction
mark CRUD event as processedcommit transaction
![Page 78: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/78.jpg)
@crichardson
ID JSON processed?
ENTITY_CRUD_EVENT
EntityCrudEventProcessor
RedisUpdater
apply(event)
Step 1 Step 2
EntityCrudEventRepository
INSERT INTO ...
Timer
SELECT ... FROM ...
Redis
![Page 79: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/79.jpg)
@crichardson
Updating Redis
WATCH restaurant:lastSeenEventId:≪restaurantId≫
Optimistic locking
Duplicate detection
lastSeenEventId = GET restaurant:lastSeenEventId:≪restaurantId≫
if (lastSeenEventId >= eventId) return;
TransactionMULTI SET restaurant:lastSeenEventId:≪restaurantId≫ eventId
... update the restaurant data...
EXEC
![Page 80: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/80.jpg)
@crichardson
Representing restaurants in Redis
restaurant:lastSeenEventId:1 55 duplicate detection
...JSON... restaurant:1
94619:Monday [0700_2:1430, 1130_1:1430, ...]
94619:Tuesday [0700_2:1430, 1130_1:1430, ...]
![Page 81: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/81.jpg)
@crichardson
Agenda
• Why polyglot persistence?
• Using Redis as a cache
• Optimizing queries using Redis materialized views
• Synchronizing MySQL and Redis
• Tracking changes to entities
![Page 82: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/82.jpg)
@crichardson
How do we generate CRUD events?
![Page 83: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/83.jpg)
@crichardson
Change tracking options
• Explicit code
• Hibernate event listener
• Service-layer aspect
• CQRS/Event-sourcing
![Page 84: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/84.jpg)
@crichardson
ID JSON processed?
ENTITY_CRUD_EVENT
EntityCrudEventRepository
HibernateEventListener
![Page 85: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/85.jpg)
@crichardson
Hibernate event listenerpublic class ChangeTrackingListener implements PostInsertEventListener, PostDeleteEventListener, PostUpdateEventListener { @Autowired private EntityCrudEventRepository entityCrudEventRepository; private void maybeTrackChange(Object entity, EntityCrudEventType eventType) { if (isTrackedEntity(entity)) { entityCrudEventRepository.add(new EntityCrudEvent(eventType, entity)); } }
@Override public void onPostInsert(PostInsertEvent event) { Object entity = event.getEntity(); maybeTrackChange(entity, EntityCrudEventType.CREATE); }
@Override public void onPostUpdate(PostUpdateEvent event) { Object entity = event.getEntity(); maybeTrackChange(entity, EntityCrudEventType.UPDATE); }
@Override public void onPostDelete(PostDeleteEvent event) { Object entity = event.getEntity(); maybeTrackChange(entity, EntityCrudEventType.DELETE); }
![Page 86: Developing polyglot persistence applications (svcc, svcc2013)](https://reader037.fdocuments.in/reader037/viewer/2022103016/554f47b4b4c905b9508b464c/html5/thumbnails/86.jpg)
@crichardson
Summary
• Each SQL/NoSQL database = set of tradeoffs
• Polyglot persistence: leverage the strengths of SQL and NoSQL databases
• Use Redis as a distributed cache
• Store denormalized data in Redis for fast querying
• Reliable database synchronization required