Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on...
Transcript of Twitter Power-User HOWTO...Visualize content and connections Ad-hoc queries Exploration Focus on...
Twitter Power-User HOWTO
@steveblackmon
✦Visualize content and connections
✦Ad-hoc queries
✦Exploration
✦Focus on specific profiles and posts
Goals
2
✦Assemble data about your twitter network
✦Analyze and Assess
✦…
✦Profit?
3
Plan
Data Visualization
4
Data Visualization
Data Repositories
5
Why ?✦ Handles de-duplication and merging
✦ Speaks JSON and REST
✦ Scales vertically and horizontally
✦ Because
6
Why ?✦ Handles de-duplication and merging
✦ Relationships central to data model
✦ Built and optimized for graph traversal queries
✦ Because
7
Supporting
Technologies
8
Why ?
✦Micro-service friendly
✦Works the same way across OS, distribution, cloud provider
✦Build, ship, run anywhere with 3-5 shell commands (build, [tag,] push, [pull,] run)
9
Why ?✦Assemble pipelines of modular data providers and database writers
✦Normalize your social databases to community standards
✦Why waste time hunting for and wrangling new SDKs?
10
11
I exist. Who am I?
Part 1: ID
12
Capture own history
Who am I?
13
Capture userstream
Who am I?
14
Other people do too.
Who are they?
15
Part 2: EGO
Collect friend/follower network
Who are they?
16
I am but one piece of a vast
society.
What is my place?
Part 3: SUPER-EGO
17
What is my place?Research interesting accounts
DEMO19
20
21
22
23
24
25
HOWTO26
✦ Using linux?
• apt-get / yum install docker.io ✦ Using a mac?
✦ Install Boot2Docker
✦ http://boot2docker.io/ • boot2docker ssh
• ifconfig
• note the bridged IP (eth1)
27
Docker
✦docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e PLUGINS=mobz/elasticsearch-head,elasticsearch/marvel/latest quay.io/steveblackmon/elasticsearch:1.4.4
✦docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch quay.io/steveblackmon/kibana:4.0.0
✦docker run -d --name neo4j -p 7474:7474 quay.io/steveblackmon/neo4j:2.2.0.M4
28
Installation
✦ docker run -d --name linkurious -p 3000:3000 —link neo4j:neo4j quay.io/steveblackmon/linkurious:latest
** LINKURIOUS IS NOT OPEN-SOURCE **** PURCHASE A LICENSE AT http://linkurio.us **
** 20% OFF IF YOU MENTION THIS TALK **** 30-DAY MONEY-BACK GUARANTEE **
** ACCESS TO THIS DOCKER REPO IF YOU BUY**
29
Installation (Cont.)
✦developer.twitter.com
✦mytwitterid.com
✦mkdir streams
✦cd streams
Configuration
(Part 0)
30
Configuration (Part 1)
twitter.userstream.json 1. Add twitter oauth credentials
2. Add own userid to twitter.follow
3. Set elasticsearch.hosts
4. Set graph.hostname
twitter.history.json 1.Add twitter oauth credentials
2.Add own userid to twitter.info
3.Set elasticsearch.hosts
✦docker run -d --name twitter.userstream -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.userstream.json -cp /twitter-0.2-SNAPSHOT.jar twitter.UserstreamPipeline
✦docker run -d --name twitter.history -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
Execution (Part 1)
32
twitter.friends.json 1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
twitter.followers.json 1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
Configuration (Part 2)
33
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 2)
34
Configuration (Part 3)Export IDs of your connections with Cypher
•MATCH (v:page {handle:'handle'})-[:follow]-(x:page) RETURN DISTINCT SUBSTRING(x.id, 11);
twitter.connections.history.json 1.Add twitter oauth credentials
2.Add connection ids to twitter.info
3.Set elasticsearch.hosts
twitter.connections.friends.json 4.Add twitter oauth credentials
5.Add connection ids to twitter.info
6.Set graph.hostname35
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline
✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 3)
36
@steveblackmon37
FIN