Genn.ai (源内)Attached realtime marketing platform

1

http://genn.ai/

Who am I?Takeshi NAKANO

Senior Researcher / Architect.Co-authered Getting started with Solr in Japanese.Co-authered Hadoop Hacks in Japanese.

2

Writing “Getting started with Kafka” in Japanese.

http://genn.ai/3

Company Name Recruit Holdings Co., Ltd

Founded March 31, 1960 (incorporated: August 26, 1963）※governance changed from Recruit Co., Ltd since October 1st, 2012

Headquarters Tokyo, Japan

Number of Group Employees 22,368（as of the end of March,2012）

Number of Group Companies 102（Consolidated Subsidiaries/as of the end of June,2012）

Capital 3,002.64 million yen（since March 1st, 1995）

Consolidated Sales 806,661 million yen （from April 1st 2011 to March 31st 2012）

Consolidated Ordinary Income

117,617 million yen（ from April 1st 2011 to March 31st 2012)

http://genn.ai/4

Recruit Holdings

HeadquartersFunction

Global Business

Business R&D

Recruit Career Co., Ltd

Recruit Jobs Co., Ltd

Recruit Sumai Company Ltd

Recruit Marketing Partners Co., Ltd

Recruit Lifestyle Co., Ltd

Recruit Administration Co., Ltd

Recruit Technologies Co., Ltd

Recruit Communications Co., Ltd

Operating Companies Domain

Recruit Staffing Co., Ltd

STAFF SERVICE HOLDINGS CO., LTD

http://genn.ai/

What is our challenge?Clients can get the information about their customers checking their items on our web sites.

Providing a printed report monthly.　　　　▽

Send them more detailed report in real time.

In real time fashion, the value of the information will be dramatically increased!

5

http://genn.ai/

Agenda

What is genn.ai?OverviewHow to gather clicks 0) Injecting our javascripts

6

Genn.ai core details Structure 1) Setting up filters 2) Filtering customers 3) Visualizing them

FutureDealing with historical streamsGenn.ai wants friends.

http://genn.ai/

Overview of using Genn.ai

7

CustomerA

CustomerB

CustomerC

Realtime Analysis Platform

Genn.ai

じゃらんWeb Sites

Client(Hotels)

MarketerAnalyst

http://genn.ai/

Demo videos!•They are based on a hotel reservation site like Expedia.com.

•An animation and a screencapture what the marketting staff see using Genn.ai.

At this moment ..•What prefectures are all my users searching for? (0:26)•Comparing the number of searches in today’s and yesterday’s regarding places. (2:50)

8

http://genn.ai/

Overview of using Genn.ai

9

CustomerA

CustomerB

CustomerC

Realtime Analysis Platform

Genn.ai

じゃらんWeb Sites

Client(Hotels)

MarketerAnalyst

http://genn.ai/

Overview of using Genn.ai

10

CustomerA

CustomerB

CustomerC

じゃらんWeb Sites

Realtime Analysis Platform

Genn.ai

Client(Hotels)

MarketerAnalyst

http://genn.ai/

How to gain access to Genn.ai

11

CustomerA

CustomerB

CustomerC

App(ex.Tomcat) Apache

Genn.ai will be able to be attached!

http://genn.ai/

How to gain access to Genn.ai

A custom apache module for embeding our code on returning html sources from App.

12

CustomerB

Apache

Genn.ai will be able to be attached!

mod

Genn.ai

App(ex.Tomcat)

http://genn.ai/

How to gain access to Genn.ai

13

The embedded values and js on a html.

http://genn.ai/

Structure

14

http://genn.ai/

Steps for getting resultsThere are 3 steps! (after getting clicks)

15

1 2 3

Pouring clicks into Kafka

Recording themas behaviors

Visualizing them

0gather clicks

Backend for Genn.ai

http://genn.ai/

Structure (Pouring into Kafka)

16

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

321

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

Frontend for Genn.ai

Visualizer for Genn.ai

Zoomdata

Backend for Genn.ai

http://genn.ai/

Structure (Recording as behaviors)

17

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Visualizer for Genn.ai

Frontend for Genn.ai

Zoomdata

Backend for Genn.ai

Visualizer for Genn.ai

http://genn.ai/

Structure (Visualizing customers)

18

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Frontend for Genn.ai

Zoomdata

http://genn.ai/

How is Genn.ai composed?Genn.ai Frontend! : Gathering clicks.Genn.ai Backend! : Recording behaviors.Genn.ai Visualizer! : Visualizing customers’ attrs.

↑Kafka : Used for message busMongoDB : Storing Additional tablesStorm : Processing click streams

19

http://genn.ai/

Details of recording behaviors.

20

Backend for Genn.ai

http://genn.ai/

Structure (Recording behaviors)

21

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Visualizer for Genn.ai

Zoomdata

Frontend for Genn.ai

http://genn.ai/

Calculated valuesRecording behaviorsUseful values for each sitesEx.)Hotel reservation site,

22

{ "_id": 154, "largeAreaCntAcl": { "Tokyo": 3, "Berlin": 1 }, "largeAreaCntPre": { "Tokyo": 4, "Berlin": 3 }, "rateAvgAcl": 4200, "rateAvgPre": 6200, "rateCntAcl": [ 4, 1, 0, 0, 0 ], "rateCntPre": [ 4, 1, 2, 0, 0 ], "lastUpdate": "2013-04-16T04:04:37.432Z"}

http://genn.ai/

Calculated valuesRecording behaviorsUseful values for each sitesEx.)Hotel reservation site,Where is the favorite place?

23

{ "_id": 154, "largeAreaCntAcl": { "Tokyo": 3, "Berlin": 1 }, "largeAreaCntPre": { "Tokyo": 4, "Berlin": 3 }, "rateAvgAcl": 4200, "rateAvgPre": 6200, "rateCntAcl": [ 4, 1, 0, 0, 0 ], "rateCntPre": [ 4, 1, 2, 0, 0 ], "lastUpdate": "2013-04-16T04:04:37.432Z"}

http://genn.ai/

Calculated valuesRecording behaviorsUseful values for each sitesEx.)Hotel reservation site,Where is the favorite place?Where is the this searching place?

24

{ "_id": 154, "largeAreaCntAcl": { "Tokyo": 3, "Berlin": 1 }, "largeAreaCntPre": { "Tokyo": 4, "Berlin": 3 }, "rateAvgAcl": 4200, "rateAvgPre": 6200, "rateCntAcl": [ 4, 1, 0, 0, 0 ], "rateCntPre": [ 4, 1, 2, 0, 0 ], "lastUpdate": "2013-04-16T04:04:37.432Z"}

http://genn.ai/

Calculated valuesRecording behaviorsUseful values for each sitesEx.)Hotel reservation site,Where is the favorite place?Where is the this searching place?What price range has he booked?

25

{ "_id": 154, "largeAreaCntAcl": { "Tokyo": 3, "Berlin": 1 }, "largeAreaCntPre": { "Tokyo": 4, "Berlin": 3 }, "rateAvgAcl": 4200, "rateAvgPre": 6200, "rateCntAcl": [ 4, 1, 0, 0, 0 ], "rateCntPre": [ 4, 1, 2, 0, 0 ], "lastUpdate": "2013-04-16T04:04:37.432Z"}

etc..

http://genn.ai/

Functions for recording behaviorsTime shift function

Time sensitive counter (old item will be expired and deleted)

Facet ( like Solr )Categorized counter

Range Facet ( like Solr )Dynamic categorized counter

26

321

http://genn.ai/

Functions for recording behaviorsTime shift function

Time sensitive counter (old item will be expired and deleted)

Facet ( like Solr )Categorized counter

Range Facet ( like Solr )Dynamic categorized counter

27

321

http://genn.ai/

Time Shift function?We make 3 min time window and now start. ( no clicks here )

28

t

Now

http://genn.ai/

Time Shift function?In our 3 min time window, 1.5 min are captured. 2 clicks have been recorded.

29

1.5 minsNow

Time window

▲Click

▲Click

2 clicks are captured.

t

http://genn.ai/

Time Shift function?There are 3 clicks in the past.The length of the capturing window becomes max.

30

3 minsNow

Time window

▲Click

▲Click

▲Click

t

http://genn.ai/

Time Shift function?There are 4 clicks in the past.Ignore the first 2 clicks

31

▲Click

▲Click

Now

Time window

▲Click

▲Click

1.5 min 3 mins

t

http://genn.ai/

Time Shift function?There are 4 clicks in the past.Ignore the first 3 clicks

32

▲Click

▲Click

Now

Time window

▲Click

▲Click

3 mins3 mins

t

http://genn.ai/

Functions for recording behaviorsTime shift function

Time sensitive counter (old item will be expired and deleted)

Facet ( like Solr )Categorized counter

Range Facet ( like Solr )Dynamic categorized counter

33

321

http://genn.ai/

Functions for recording behaviorsTime shift function

Time sensitive counter (old item will be expired and deleted)

Facet ( like Solr )Categorized counter

Range Facet ( like Solr )Dynamic categorized counter

34

321

http://genn.ai/

Counting the frequencyThis function is under the effect of the Time shift function.

35

{ "_id": 154, "largeAreaCntAcl": { "Tokyo": 3, "Berlin": 1 }, "largeAreaCntPre": { "Tokyo": 4, "Berlin": 3 }, "rateAvgAcl": 4200, "rateAvgPre": 6200, "rateCntAcl": [ 4, 1, 0, 0, 0 ], "rateCntPre": [ 4, 1, 2, 0, 0 ], "lastUpdate": "2013-04-16T04:04:37.432Z"}

http://genn.ai/

Counting the frequencyThis function is under the effect of the Time shift function.You have booked 1 stay in Berlin and 3 stays in Tokyo in this 3 weeks.

36

▲L.A.

▲S.F.

t

Now

Time window

▲Berlin

▲Tokyo

1.5 week 3 weeks

▲Tokyo "largeAreaCntAcl": {

"Tokyo": 3, "Berlin": 1}

▲Tokyo

http://genn.ai/

Details of visualization.

37

Backend for Genn.ai

http://genn.ai/

Visualizing the customers

38

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Visualizer for Genn.ai

Backend for Genn.ai

http://genn.ai/

Visualizing the customers

39

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Visualizer for Genn.ai

http://genn.ai/

Kafka consumerKakan.puThe program for relaying clicks from Kafka to the other systems.

The enriching function (joining the MongoDB data)The converting function (for following systems)

40

Backend for Genn.ai

http://genn.ai/

Visualizing the customers

41

StormTopology

(Filter)

Kafka

kakan.puTomcatiPad(APP)

MongoDB

Genn.ai Front(Based on Finagle)

Querygenn.ai/web)Click Streams

321

Visualizer for Genn.ai

Zoomdata

http://genn.ai/

Visualizing toolZoomdataA product containing a set of server app and ipad app for visualizing event data.

42

321

http://genn.ai/

ZoomdataZoomdata has the functions like..WebAPI for streaming data.Own event processor for vizualizing.

43

321

The bar and bubble graph on default setting.Customizable visualizing templates.

http://genn.ai/

Feature plan.

44

http://genn.ai/

Genn.ai EvolutionAdding functions on Genn.ai core.Genn.ai should have filtering / aggrigation functions for federating with other systems.

Historical rewinding function will be implemented with the Hadoop cluster.

45

http://genn.ai/

Future planGenn.ai wants friends!

We are looking for co-challenging partners for tuning and improving Genn.ai.

You can use Genn.ai on your web sites for totally free (infrastructure etc..) if we could share the dream.

Genn.ai codes will be opensourced in the future.

Please feel free to contact us!http://genn.ai/

46

http://genn.ai/

Thanks!

47

48