Presented By: Ashu Raj09/21/2010
CSE 6339 DATA EXPLORATION AND ANALYSIS IN RELATIONAL DATABASES FALL 2010
Problem Statement Application Examples Bayesian Method
The learning PhaseThe Characterization PhaseThe Inference Phase
Experiments and Results Conclusion
A ubiquitous problem in data management:
Given a finite set of real values, can we take a sample from the set, and use the sample to predict the kth largest value in the entire set?
Min/Max online aggregation Top-k query processing Outlier detection Probability Query Optimization
Propose a natural estimator
Characterize error distribution of estimator (Bayesian)Learn a prior model from the past query
workloadUpdate the prior model using a sampleSample an error distribution from the posterior
model
With estimator and its error distribution, we can confidence bound the kth largest
Data set size N Sample size n Estimator is the (k´)th largest in sample
k´/n = k/N
So, k´= ┌ (n/N * k) ┐
How accurate this method is?
How to determine the estimator’s error?
Study the relationship (estimator vs answer)
Take the ratio of k th and (k´)th and find the ratio distribution
Don’t have any prior knowledge of D. It is impossible to predict the ratio by looking
at the sample only.
With domain knowledge and sample, we can guess behavior of D
What domain knowledge should be modeled to help solving this problem?
Setup: four data sets with different histogram shapes; each has10,000 values; we are looking for the largest value.
Experiment: take a 100-element sample, record the obtained ratio kth/(k´)th . Do this 500 times.
Importance of query shape
Proposed Bayesian Inference Framework
Learning PhaseCharacterization PhaseInference Phase
The Generative Model Assume the existence of a set of possible shape patterns Each shape has a weight, specifying how likely it matches with a new
data set’s histogram shape.
First, A biased die is rolled to determine by which shape pattern the query result set will be generated.(in previous figure, suppose we select shape 3)
Next, Arbitrary scale for the query is randomly generated.
Instantiate a parametric distribution f(x│shape,scale); this distribution is repeatedly sampled from to generate the new data set.
Next, formalize and learn the model from the domain data(workload)
Probability Density Function(PDF)Gamma distribution can produce data with
arbitrary right leaning skew.
Deriving the PDF:Gamma Distribution PDF is:
Where α > 0,known as the shape parameter and β > 0, known as inverse scale parameter.
Since scale does not matter, we treat β as an unknown random variable.
Deriving the likelihood Model:The resulting likelihood of a given dataset D is in
the form:L(D│α)
This model assumes that a set of c weighted shapes. So, the complete likelihood model of observing D is:
Where wjs are each non-negative weights and ∑cj=1
wj =1.
The complete set of model parameters is ⱷ = {ⱷ1,ⱷ2,…,ⱷc} where ⱷj = {wj, αj}
Learning the parameters: ⱷ is unknown and must be learned from the
historical workload.Given a set of independent domain data
sets D ={D1, . . . ,Dr }, the likelihood of observing them is:
We use EM algorithm to learn the most likely so that:
At this point , We have learned a prior shape model
Now we have to apply EM algorithm to this prior modelNow we take the sample from the data set
Use the Sample to update prior weight of each shape pattern
EM Algorithm:
Let S be our sample, applying Baye’s rule, the posterior weight of shape pattern j is:
The resulting posterior shape model:
Derive error distribution associated with each shape.Each shape characterizes an error distribution
of kth/(k´)thTo find the error distribution for α,
Pick a scale β1. Query is produced by drawing a sample size N from the
distribution Gamma f(x│α , β),the kth largest value in this sample is f(k).
2.In order to estimate f(k), a sub sample of size n is drawn from the Sample obtained in step 1. the (k ´)th largest value in the subsample is the estimator f(k)´.
Monte- Carlo Sampling
TKD (Top k dependent) METHOD efficiently produce f(k)´ given f(k). First determines whether or not the subsample includes f(k)
by means of a Bernoulli trail.Depending upon the result, The TKD method figures out in
the randomized method, the composition of the k´ largest values in the subsample with the help of Hypergeometric Method and returns the (k´)th largest.
The input parameters are same as Monte Carlo method with the addition of the sampled f(k).
This process assumes that we have an efficient method to sample a f(k) efficiently.
Each shape characterizes an error distribution kth/(k´)th
To get the posterior error distribution, attach each shape’s posterior weight to it’s error distribution.
The final mixture error distribution:
Given the distribution of kth/(k´)th, we can confidence bound the answer:Choose pair of lower bound and upper
bound (LB,UB),such that p% probabilty is covered.
Bound kth by [(k´)th * LB , (k´)th * UB] with p% probability.
Learn a prior shape model from historical queriesDevised a close-form model: a variant of Gamma
mixture modelEmployed an EM algorithm to learn the model from
historical data Update prior shape model with a sample
Applied Baye’s rule to update shape pattern’s weight
Produce an error distribution from the posterior modelPosterior weight attached to each shape’s error
distribution With our estimator and its error distribution, we can
bound answer.
Distance-Based Outlier Detection
Improve the performance of state-of-the-art algorithm on an average factor of 4 over seven large data sets.
Defined the problem of estimating the kth largest value in a real data set.
Proposed an estimator Characterized the ratio error
distribution by a Bayesian framework.
Applied the proposed method to research problems successfully.
Mingxi Wu, Chris Jermaine. A Bayesian Method for Guessing the Extreme Values in a Data Set , VLDB 2007.
http://www.cise.ufl.edu/~mwu/research/extremeTalk.pdf
Top Related