Abstracting with pacer

© All Rights Reserved 2013 | xnlogic.com

Abstracting�� your�� Graphwith�� Pacer


Querying�� Twitter


How would you query Twitter?

Find conversations that are relevant to me, happened recently and were popular.


��

person.recent_popular_relevant_conversations!


Layer�� 1:�� Graph�� Foundations


It’s all about moving through the graph and filtering out anything you are not interested in


Moving�� Through�� The�� Graph

Follow an edge pointing out of a vertex

out_e.in_v

or one pointing in to my vertex

in_e.out_v


These operations are fast


Filtering�� for�� Understanding


Most�� Fundamental:�� By�� Edge�� Label

Who I follow

out_e(:follow).in_v Who follows me

in_e(:follow).out_v

follow


By�� Property

out_e.in_v(name: 'elonmusk')

filter(name: 'elonmusk')

name: 'elonmusk'


Layer�� 2:�� Building�� Meaning


Vertex�� Types�� in�� Your�� Domain

g.v type: "person"

Pacer just uses Ruby’s built-in modules as its types

module Person def self.lookup { type: "person" } end end

g.v(Person)


Name�� Basic�� Movements�� From�� a�� Person

module Person::Route

def following out_e(:follow).in_v(Person) end

def followed_by in_e(:follow).out_v(Person) end

end

Person

follow

Person


Moving�� From�� One�� Type�� To�� Another

module Person::Route

def tweets out_e(:wrote).in_v(Tweet) end

end

Person Tweet

wrote


Aside:�� Using�� Our�� New�� Definitions

graph = Pacer.neo4j(‘path/to/graph’) #=> <PacerGraph>

graph.v(Person).tweets # <V[123] Tweet> <V[321] Tweet> # <V[456] Tweet> #=> <Route Index -> out(:tweet)>

It’s really just that simple.


Basic�� Domain�� Filters

def female_only filter(gender: ‘female’) end

def recent_only where(‘time > :recent’, recent: hours_ago(2)) end

def max_length_only filter do |tweet| tweet[:text].length == 140 end end


These methods are extremely simple.

Q: Why even bother with them?

A: Names, names, names


Layer�� 3:�� Chains�� of�� Thought


Meaningful�� Movements

def retweeted_by retweets.authors end

def current_timeline following.tweets.recent_only end


More�� Powerful�� Filters:�� Lookaheads

Person followed by at least 1000 people

def popular_only lookahead(min: 1000) { |person| person.followed_by } end

Tweet retweeted at least 100 times

def impactful_only lookahead(min: 100) { |tweet| tweet.retweets } end


It’s�� Even�� Better:�� Filtered�� Lookaheads

Have these people tweeted recently?

def active_only lookahead { |person| person.tweets.recent_only } end

Reversing a filter (lookahead Inception!)

def inactive_only neg_lookahead { |person| person.active_only } end


Layer�� 4:�� Composition


Extracting�� Deeper�� Meaning

A shy twitterer retweets and stars stuff, follows people but never tweets:

def shy_people_only has_retweeted. has_starred. is_following. neg_lookahead(&has_tweeted) end


Reality�� Folds�� Back�� On�� Itself

We are marking an element and circling back to it

def relevant_conversation_tweets as(:me).following.tweets.lookahead do |tweet| tweet.responded_to.authors.followed_by.is(:me) end end

This pattern is incredibly powerful!


Exercise:�� How�� Many�� Lines�� Of�� Code?

How did we build such a sophisticated query out of such simple pieces?

def recent_popular_relevant_conversations! recent_only.popular_only.relevant_conversation_tweets end


These methods are extremely simple.

Q: Why even bother with them?

A: Names, names, names


You’ve just isolated your logic from the schema.

Most of your business logic is at Layer 4+.

Even major schema changes rarely impact beyond Layer 3.


Layer�� 5:�� Repeating�� Patterns


Some�� Patterns�� are�� Variable�� Length

The tweet this responded to, then the tweet that it responded to, etc, etc...

def conversations all { |tweet| tweet.responded_to } end

Any pattern can be repeated, no matter how complex

def popular_conversations all { |tweet| tweet.responded_to.popular_only } end


Ignoring�� Intermediate�� Vertices

The tweet that started it all

def conversation_starters deepest { |tweet| tweet.responded_to } end


Combining�� Combination�� Combinations

A lookahead with a loop with a lookahead, emerging with a loop.

def conversations_with_a_recent_popular_exchange lookahead do |tweet| tweet.popular_conversations.recent_only end.conversations end


Specialized�� Loop�� Logic

Make common operations simple, uncommon operations possible... def rapid_conversations loop do |tweet| tweet.responded_to end.while do |tweet, depth, path| if depth >= 1 prev_tweet = path[-2] minutes = tweet.minutes_after(prev_tweet) if minutes < 2 :loop_and_emit if minutes < 5 :loop end else :loop end end end


Sometimes the data is like this.Sometimes the data is like that...

In the real world, data isn’t so clean. We’ve got you covered.


Branching�� and�� Merging

You’ve got the power to encode alternate routes

def good_tweets branch do |person| person.starred_tweets end.branch do |person| person.tweets.retweeted end end

That’s often all it takes to handle messy data from multiple sources.


What�� Else?


What�� else?

• Transactional CRUD is easy• Indices are automatic• Summarization tools• Data export (for visualization, etc)• Process more than just graphs

Abstracting with pacer

Technology

Transcript of Abstracting with pacer