Graph Adoption at Gamesys - Toby O'Rourke @ GraphConnect SF 2013
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
-
Upload
neo4j-the-fastest-and-most-scalable-native-graph-database -
Category
Technology
-
view
684 -
download
0
description
Transcript of GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
BETTING THE COMPANY(AGAIN?!) ON A
GRAPH DATABASETHE STORY CONTINUED…
Aseem KishoreOct 2014
MIX GRAPHSBRINGING IDEAS TOGETHER
MIX NEO4J
MIX GRAPHS
STREAMSMATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)
RETURN creation
PAGINATION (BAD)MATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)
RETURN creationORDER BY creation.createdAt DESCSKIP {count} * {page - 1}LIMIT {count}
PAGINATION (GOOD)MATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)
WHERE creation.createdAt < {cursorTime}RETURN creationORDER BY creation.createdAt DESCLIMIT {count}
REMIX FAMILIESMATCH (c:Creation {id: {id}})MATCH (c) -[:remix_source*0..]- (relative)
WHERE relative.createdAt < {cursorTime}RETURN relativeORDER BY relative.createdAt DESCLIMIT {count}
HOME STREAM 1MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) <-[:creator]- (creation)
WHERE creation.createdAt < {cursorTime}RETURN creationORDER BY creation.createdAt DESCLIMIT {count}
HOME STREAM 2MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)
WITH creation, starORDER BY star.createdAtWITH creation, HEAD(COLLECT(star)) AS star
WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}
HOME STREAM 3MATCH (me:User {id: {id}})MATCH (me) -[:starred]-> (c) <-[:remix_source*]- (remix)
WHERE remix.createdAt < {cursorTime}RETURN DISTINCT remixORDER BY remix.createdAt DESCLIMIT {count}
UNION?
UNTIL THEN…nodes = _(results).chain().flatten() .sortBy (node) -> node._orderedAt .unique (node) -> node.id .reverse().value()
Post-processing on our server.
DEDUPING (VERY BAD)MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)
WITH me, creation, starORDER BY star.createdAtWITH me, creation, HEAD(COLLECT(star)) AS star
MATCH (creation) -[:creator]-> (creator)WHERE NOT (me) -[:follows*0..1]-> (creator)
WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}
DEDUPING (BAD)MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)
WITH me, creation, starORDER BY star.createdAtWITH me, creation, HEAD(COLLECT(star)) AS star
MATCH (creation) -[:creator]-> (creator)WHERE creator <> me AND NOT((me) -[:follows]-> (creator))
WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}
QUERY PROFILINGfor key, query of queries echo "Query '#{key}':"
# warm-up: neo4j.query query, params, _
times = [] for i in [1..3] start = Date.now() neo4j.query query, params, _ times.push Date.now() - start
# ... echo "Min/median/max: #{min}/#{median}/#{max} ms. Mean: #{Math.round mean} ms."
(Hat-tip Mark Needham)
HOME STREAM0-following-ids 27 ms1-following-shares 581 ms2-following-features 77 ms3-following-stars 1386 ms4-stars-remixes 189 ms5-shares-remixes 81 ms
All in parallel 1961 ms
(On my aging MacBook Air, for our ~worst-case user.)
IN PRODUCTION…
(But still some || mystery to unravel…)
THRESHOLD
THRESHOLDMATCH (me:User {id: {id}})
WITH me, TOFLOAT(CASE WHEN me.numFollowing < 1 THEN 1 ELSE me.numFollowing END) AS ̀me.numFollowing̀WITH me, FLOOR(LOG(3 * ̀me.numFollowing̀ / 100) / LOG(3)) AS thresholdWITH me, (CASE WHEN threshold < 0 THEN 0 ELSE TOINT(threshold) END) + 1 AS threshold
MATCH (me) -[:follows]-> (following) -[star:starred]-> (creation)
WITH creation, star, thresholdORDER BY star.createdAtWITH creation, COLLECT(star) AS stars, thresholdWHERE LENGTH(stars) >= thresholdWITH creation, stars[threshold - 1] AS star
WITH creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}
MATCH (creation) -[:creator]-> (creator)RETURN creation, creator, _starredAt
THANK YOU