Post on 15-Apr-2017
Smart Personal Shopping Assistant (SPA)
Arvind Rapaka
Sairam BantupalliRavindra Nath
SpotDy Incwww.spotdy.com
Why we need?
A Personal Shopping Assistant is an occupation where people help customers by giving advice and making suggestions. They are employed by departmental Stores.
But you have a Mobile/Web Ecommerce business. How can you enable your mobile/web application turn into a smart personal shopper for your customers? Enter - Smart Personal Shopping Assistant.
Customer
StoreAssistant
Online Store
Why we need ?
I need skinny pants that girls like. My size is 32 inch waist and 34 length.
Here you go. Let me know If I should filter by price, size or brand
I like it. My price range is 40-50 dollars.
I need skinny pants that girls like. 32 inch waist, 34 length.
I like this pant. Let’s buy it.
Ok, I placed the order. You should receive your order by tomorrow. Best of luck.
Why we need ?
Can you place an order of red skinny pants that I ordered last year
Do you want the same size?
Yes
I have placed the order. You should receive your pants by tomorrow.
Process Overview
ASR Image Q&A
Knowledge Graph/ Image DB
● Speech Recognition ● Image Matching● Q & A Dialogue
ASR- Acoustic Speech RecognitionQ&A - Question and Answers Dialogue
SPA - System Call Flow
Q&A Dialogue and IR
ASR
Image Analysis Engine
Pre-computed KD
SpotDy BigAITM Platform
Image
Text
Voice
IR - Information RetrievalKD - Knowledge Graph DB
Dialogue/Action
Dialogue/Action
Knowledge Database (KD)
● Build Product Knowledge Database○ Classification (LDA, Existing Taxonomy)
○ NLP Analysis (CRF, Bayesian etc ..)
○ Image Analysis/ Text Attribution (SURF)
○ Ontologies
Image Matching
● Image Analysis ○ Extract Feature (SURF Feature Extraction)
■ Find keypoints
○ Grouping Descriptors (SURF Feature Descriptor)■ Keypoints are grouped in descriptors
○ Match images in the precomputed descriptor database.
○ Post Processing
ASR
● Speech Recognition ○ Extract Feature vectors
○ Speech Decoder■ Scoring (DNN)
■ Most Likely Text from Acoustic Model (HMM/Viterbi Algorithm)
○ Pass to Q & A system
Question and Answering
● Query Analysis ○ Query Processing (Stemming, lemma, Gazetteer ..)
○ Understand User Intent (HMM )■ Navigational/Specificity
■ Initiate dialogue if necessary
○ Query POS Labelling / Entity Extraction (CRF)
○ Query Rewrite/ Retrieval/Post Processing
Q&A Personalization
● Q & A results should be personalized and
aggregated based on:
○ Past user history
○ User Geo/Demo
○ Occasions such as Christmas, Thanksgiving etc ..
SpotDy BigAITM
Query Results
SURF (Speeded up Robust Features)
SURF is a feature detection process to examine an image to extract features, that are unique to the objects in the image. Based on SIFT but faster.
In our case, it help in retrieving similar products based on images.
Process Involves : ○ Build Scale Space○ LoG Approximation○ Key Point Extraction○ Generate Features
LoG Approximation
● The Laplacian is a 2-D isotropic measure of the 2nd spatial derivative of an image.
● The Laplacian of an image highlights regions of rapid intensity change to detect edges.
● Uses Gaussian smoothing filter in order to reduce its sensitivity to noise due to second derivation
GMM/DNN-HMM
HMM is a generative probabilistic model that provides a framework for modelling time-varying spectral vector sequences. In our case, we use for speech recognition.
● GMM/DNN produce posterior probabilities for HMM States● predicts likelihood of observation sequence being generated by
state sequence using Viterbi Algo● Sub word HMMs concatenate to create larger word-based HMM
Observations (Feature vectors)
GMM/DNN
HMM States(Senones)
Posterior Probabilities
NLP
Knowledge Database (KD) is the key for the query processing and information retrieval
● NLP is extensively used to process unstructured data in building KD.
Algorithms:● Conditional Random Fields/Maxent for POS Tagging, Entity
Extraction, concept tagging etc.● LDA for topic Analysis and Classification
Q&A Dialogue and IR
IndexedKD
Product Catalog
Product metainfo
NLP Engine
Query Processing
Query
Indexed KD
Annotators/Filters
Results
● User Query pass goes through various annotators. Some of the few annotators include :
○ Gazetteer, Lemmatization, Stemming, POS Tagging, Entity Extraction
● Query Rewrite ● Search - Similarity (IR). Basic
Algorithms include ○ Vector Space Modelling○ BM25F
● Result Generation
SPA - HA Architecture
Significant computing resources are required while
scaling to millions of requests in real time.
BigAITM
BigAITM is purpose built for the scalability of applications such as SPA.
● Building KD (Knowledge Database)
● Image Repository Store
● Query Processing
● Scalable Machine Learning Models