Bump Hunting
The objectivePRIM algorithmBeam search
References:Feelders, A.J. (2002). Rule induction by bump hunting. In J. Meij (Ed.), Dealing with the data flood (STT, 65) (pp. 697-700). Den Haag, the Netherlands: STT/Beweton.J.H. Friedman and N.I. Fisher (1999) Bump-hunting in high-dimensional data. Statistics and Computing, 9:123–143.
Brief Intro to UndirectedGraphical models
OverviewRegression-based models
Bump Hunting - The objective
Find regions in the feature space, where the outcome variable has high average value.
In classification, it means a region of the feature space where the majority of the samples are in one class.
The decision rule looks like an intersection of several conditions (each on one predictor variable)
If condition 1 & condition 2 &…… & condition N, then predict value …
Ex: if 0<x1<1 & 2<x2<5 &…& -1<xn<0, then class 1
Bump Hunting - The objective
When the dimension is high, and there is many such boxes, the problem is not easy.
Bump Hunting - The objectiveLet’s formalize the problem:Predictors x=( )Target variable y, either continuous or binary
Feature space:
Find subspace such that
Note: when y is binary, this is not mean of y. Rather, it is Pr(y=1 | x R)
Define any box:
Box in continuous feature space:
Bump Hunting - The objective
Box in categorical feature space.
Bump Hunting - The objective
Sequentially find box in subsets of the data.
Bump Hunting - PRIM
Support of a box:
N
iiB BxI
N 1
)(1
Continue search for boxes until not enough support for the new box.
Bump Hunting - PRIM
“Patient Rule Induction Method”
Two steps:
(1) Patient successive top-down refinement(2) Bottom-up recursive expansion
These are greedy algorithms.
Bump Hunting - PRIMPeeling:Begin with box B containing all data (or all remaining data in later steps)Remove sub-box b*, which maximizes in B-b*The candidate box b is defined on a single variable (peeling only in one of the dimensions), and only a small percentile is peeled each time.
Bump Hunting - PRIM
This is a greedy hill-climb algorithm.Stop the iteration when the support drops to pre-determined threshold.Why called “patient …”? Only remove a small fraction at each step.
Bump Hunting - PRIM
Pasting:
In peeling, box boundries are determined without knowledge of later peels. Some non-optimal steps can be taken.Final box could be improved by boundary adjustments.
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
Bump Hunting - PRIM example
2/7
Bump Hunting - PRIM example
Bump Hunting - PRIM example
3/7
Bump Hunting - PRIM exampleThe winner is:
Bump Hunting - PRIM exampleThe next peel:
1. And β= 0.4
Bump Hunting - PRIM example
Bump Hunting - Beam search algorithm
At each step, w best sub-boxes (each on a single variable) are selected.Minimum support requirement.More greedy --- at each step, much more can be peeled than PRIM optimization on one of the variables.
Bump Hunting - Beam search algorithm
W=2
Bump Hunting - Beam search algorithm
Bump Hunting - Beam search algorithm
Bump Hunting - About PRIM
It is a greedy search. However, it is “patient”. This is important.
Methods that partition the data much faster, e.g. Beam search and CART, could be less successful.
The “patient” method makes it easier to recover from previous “unfortunate” steps, since we don’t run out of the data too fast.
It doesn’t select off predictors due to high correlation within them.
Undirected Graph Models - Introduction
A network/graph is a set of vertices connected by edges.undirected edges “undirected network”directed edges “directed network”.
Vertex-level characteristic:The number of connections to a vertex : “degree”
Incoming edges “in-degree” ki
Outgoing edges “out-degree” ko
k=ki+ko
ki ko
Evolution of networks. S.N. Dorogovtsev, J.F.F. Mendes
Undirected Graph Models - Introduction
Graphical models – a visual expression of the joint distribution of the entire set of random variables.
Undirected graphical model – also known as “Markov random fields” or “Markov networks”.
Lack of connection in such a network – conditional independence given all other variables.
Sparse graphs – small number of edges – easy to interpret.
Edges – encode the strength of conditional dependency.
Undirected Graph Models - Introduction
Undirected Graph Models - Introduction
Y “separates” X and Z
Pairwise Markov independency
Ex:
Global Markov independency:Subgraphs A, B and C. If every path between A and
B intersects with a node in C C separates A and B.
Ex:
Undirected Graph Models - Introduction
Pairwise Markov independency
Based on Global Markov independency
Clique – a complete (all pairs connected) subgraphMaximal clique – a clique; no other vertices can be added to yield a clique.
Ex: {X, Y}, {Y, Z}, {Z, W} of graph above
Undirected Graph Models - Introduction
A probability density function f over a Markov graph G can be presented:
Either distribution can represent the dependence structure:
Pairwise Markov graphs concerns f(2) above.
Undirected Graph Models – Gaussian Graphical Model
- Observations have a multivariate Gaussian distribution with mean μ and covariance matrix Σ.
- Gaussian distribution represents at most second-order relationships, it automatically encodes a pairwise Markov graph.
- All conditional distributions are also Gaussian. - If the ijth component of Θ = Σ−1 is zero, then variables i
and j are conditionally independent - Y is one variable, Z = (X1,...,Xp−1) is the rest of variables,
then the conditional distribution is
Same as population multiple linear regression of Y on Z
Partition Θ = Σ−1 the same way, then because ΣΘ = I,
Thus the regression coefficient of Y~Z,
- Zero elements in β and hence θZY mean that the corresponding elements of Z are conditionally independent of Y
- We can learn the dependence structure through multiple linear regression.
Undirected Graph Models – Gaussian Graphical Model
Finding parameters when network structure is known.
- Take empirical mean x and ̄� covariance matrix
- The log likelihood of data is
- The quantity −l(Θ) is a convex function of Θ
Undirected Graph Models – Gaussian Graphical Model
Undirected Graph Models – Gaussian Graphical Model
Estimating graph structure:
Meinshausen and Bu hlmann’s regression approach ̄�(2006)
- Fit a lasso regression using each variable as the response and the others as predictors.
- θij is estimated to be nonzero if estimated coefficient of variable i on j is
nonzero,OR (alternatively AND)
estimated coefficient of variable j on i is nonzero
Undirected Graph Models – Gaussian Graphical Model
More formally – graphical lasso- Penalized likelihood
Top Related