Basic Statistics and Shannon Entropy Ka-Lok Ng Asia University.
-
date post
22-Dec-2015 -
Category
Documents
-
view
220 -
download
1
Transcript of Basic Statistics and Shannon Entropy Ka-Lok Ng Asia University.
Mean and Standard Deviation (SD)
• Compare distributions having the same mean value– a small SD value a narrow distribution– a large SD value a wide distribution
Pearson correlation coefficient or Covariance
Statistics – standard deviation and variance, var(X)=s2, for 1-dimension data
How about higher dimension data ? - It is useful to have a similar measure to find out how much the dimensions vary from the mean with respect to each other. - Covariance is measured between 2 dimensions,- suppose one have a 3-dimension data set (X,Y,Z), then one can calculate Cov(X,Y), Cov(X,Z) and Cov(Y,Z)
- to compare heterogenous pairs of variables, define the correlation coefficient or Pearson correlation coefficient, -1≦ XY ≦1
))(var(var
),(
YX
YXCovXY -1 perfect anticorrelation
0 independent+1 perfect correlation
1
)()( 1
22
n
xxsxVar
n
i i
1
))((),( 1
n
yyxxYXCov
n
i ii
The squared Pearson correlation coefficient
• Pearson correlation coefficient is useful for examining correlations in the data
• One may imagine an instance, for example, in which an event can cause both enhancement and repression.
• A better alternative is the squared Pearson correlation coefficient (pcc),
)var()var(
)],([ 22
YX
YXCovXYsq
The square pcc takes the values in the range 0 ≦ sq 1.≦0 uncorrelate vector1 perfectly correlated or anti-correlated
pcc are measures of similaritySimilarity and distance have a reciprocal relationshipsimilarity↑ distance↓ d = 1 – is typically used as a measure of distance
Pearson correlation coefficient or Covariance
- The resulting XY value will be larger than 0 if a and b tend to increase together, below 0 if they tend to decrease together, and 0 if they are independent.Remark: XY only test whether there is a linear dependence, Y=aX+b- if two variables independent low XY, - a low XY may or may not independent, it may be a non-linear relation, for example, y=sin x)- a high XY is a sufficient but not necessary condition for variable dependence
Pearson correlation coefficient
• To test for a non-linear relation among the data, one could make a transformation by variables substitution
• Suppose one wants to test the relation u(v) = avn
• Take logarithm on both sides• log u = log a + n log v • Set Y = log u, b = log a, and X = log v a linear relation, Y = b + nX log u correlates (n>0) or anti-correlates (n<0) w
ith log v
Pearson correlation coefficient or Covariance matrixA covariance matrix is merely collection of many covariances in the for
m of a d x d matrix:
Spearman’s rank correlation
• One of the problems with using the PCC is that it is susceptible to being skewed by outliers: a single data point can result in situation appearing to be correlated, even when all the other data points suggest that they are not.
• Spearman’s rank correlation (SRC) is a non-parametric measure of correlation that is robust to outliers.
• SRC is a measure that ignores the magnitude of the changes. The idea of the rank correlation is to transform the original values into ranks, and then to compute the correlation between the series of ranks.
• First we order the values of gene A and B in ascending order, and assign the lowest value with rank 1. The SRC between A and B is defined as the PCC between ranked A and B.
• In case of ties assign midranks both are ranked 5, then assign a rank of 5.5
Spearman’s rank correlation
n
i i
n
i i
n
i iiSRC
yyxx
yyxxYX
1
2
1
2
1
])(][)([
))((),(
The SRC can be calculated by the following formula, where xi and yi denote the rank of the x and y respectively.
An approximate formula in case of ties is given by
)1(
)(61),(
21
2
nn
yxYX
n
i iiSRC
Distances in discretized space
• Sometimes one has to due with discreteized values
• The similarity between two discretized vectors can be measured by the notion of Shannon entropy.
Entropy and the Second Law of Thermodynamics: Disorder and the Unavailability of energy
Ice melts, it becomes more disordered and less structured.
Entropy alwaysincrease
Statistical Interpretation of Entropy and the Second Law S = k ln S = entropy, k = Boltzmann constant, ln = natural logarithm of the number of microstates corresponding to the given macrostate.
L. Boltzmann (1844-1906)
http://automatix.physik.uos.de/~jgemmer/hintergrund_en.html
Concept of entropy
Toss 5 coins, outcome
5H0T 14H1T 5
3H2T 10 2H3T 101H4T 50H5T 1
A total of 32 microstates. Propose entropy, S ~ no. of microstates, i.e. S ~
Generate coin toss with Excel
The most probable microstates
Shannon entropy
• Shannon entropy is related to physical entropy• Shannon ask the question “What is information ?”• Energy is defined as the capacity to do work, not the work itself. Work is a fo
rm of energy.• Define information as the capacity to store and transmit meaning or knowled
ge, not the meaning or knowledge itself.• For example, a lot of information from WWW, but it does not mean knowled
ge• Shannon suggest entropy is the measure of this capacitySummary
Define information capacity to store knowlege entropy is the measure Shannon entropy
Entropy ~ randomness ~ measure of capacity to store and transmit knowledge
Reference: Gatlin L.L, Information theory and the living system, Columbia University Press, New York, 1972.
Shannon entropy
• How to relate randomness and measure of this capacity ?
Microstates5H0T 14H1T 53H2T 10 2H3T 101H4T 50H5T 1
Physical entropyS = k ln
Shannon entropy
Assuming equal probability of each individual microstate, pi
pi = 1/
S = - k ln pi
Information ~ 1/pi =
If pi = 1 there is no information, because it means certainty
If pi << 1 there are more information, that is information is a decrease in uncertainty
Distances in discretized space
n
i ii ppH11 log
Sometimes it is advantageous to use a discretized expression matrix as the starting point, e.g. to assign values 0 (expression unchanged, 1 (expression increased) and -1 (expression decreased).The similarity between two discretized vectors can be measured by the notion of Shannon entropy.Shannon entropy, H1
-Probability of observing a particular symbol or event, pi, with in a given sequence
Consider a binary system, an element X has two states, 0 or 1
base 2References: 1. http://www.cs.unm.edu/~patrik/networks/PSB99/genetutorial.pdf2. http://www.smi.stanford.edu/projects/helix/psb98/liang.pdf3. plus.maths.org/issue23/ features/data/
Claude Shannon
- father of information theory
- H1 measure the “uncertainty” of a probability distribution- Expectation (average) value of information
11
n
i ipand
Shannon Entropy
pi
1,0 -1*1[log2(1)] = 0
½, ½ -2*1/2[log2 (1/2)]=1
22-states 1/4 -4*1/4[log2(1/4)]=2
2N-states
1/2N
-2N*1/2N[log2(2N)]=N
ii
i pp 2log
Uniform probability
certain No informationMaximal value
DNA seq. n = 4 states, maximum H1 = - 4*(1/4)* log(1/4) = 2 bits
Protein seq. n = 20 states, maximum = - 20*(1/20)*log(1/20) = 4.322 bits, which is between 4 and 5 bits.
The Divergence from equi-probability
• When all letter are equi-probable, pi = 1/n• H1 = log2 (n) the maximum value H1 can take• Define Hmax
1 = log2 (n) • Define the divergence from this equi-probable state, D1
• D1 = Hmax1 - H1
• D1 tells us how much of the total divergence from the maximum entropy state is due to the divergence of the base composition from a uniform distribution
For example, E. coli genome has no divergence from equi-probability because H1
Ec = 2 bits, but, for M. lysodeikticus genome, H1
Ml = 1.87, then D1 = 2.00 – 1.87 = 0.13 bit
Divergence from independenceSingle-letter events which contains no information about
how these letters are arranged in a linear sequence
Hmax1
H1 = log2 (n)
D1
Divergence from independence – Conditional Entropy
QuestionDoes the occurrence of any one base along the DNA seq. alter the pro
bability of occurrence of the base next to it ? What are the numerical values of the conditional probabilities ? p(X|Y) = prob. of event X condition on event Y p(A|A), p(T|A), p(C|A), p(T|A) … etc. If they were independent, p(A|A) = p(A), p(T|A) = p(T) …. Extreme ordering case, equi-probable seq., AAAA…TTTT…CCCC
…GGGG… p(A|A) is very high, p(T|A) is very low, p(C|A) = 0, p(G|A) = 0 Extreme case, ATCGATCGATCG…. Here p(T|A) = p(C|T) = p(G|C) = p(A|G) = 1, and all others are 0 Equi-probable state ≠ independent events
Divergence from independence – Conditional Entropy
• Consider the space of DNA dimers (nearest neighbor)• S2 = {AA, AT, …. TT}• Entropy of S2, H2 = -[p(AA)log p(AA) + p(AT) log p(AT) + …. + p(TT)
log(TT)]• If the single letter events are independent, p(X|Y) = p(X),• If the dimer event is independent, p(A|A)=p(A)p(A), p(A|T)=p(A)p(T),
…. • If the dimer is not independent, p(XY) = p(X)p(Y|X), such as p(AA) =
p(A)p(A|A), p(AT) = p(A) p(T|A) … etc.• HInp
2 = entropy of completely independent• Divergence from independence, D2 = HInp
2 – H2
• D1 + D2 = the total divergence from the maximum entropy state
Divergence from independence – Conditional Entropy
• Calculate D1 and D2 for M. phlei DNA, where p(A)=0.164, p(T)=0.162, p(C)=0.337, p(G)=0.337.
• H1= -(0.164 log 0.164 + 0.162 log 0.162 + ..) = 1.910 bits
• D1 = 2.000 – 1.910 = 0.090 bit
• See the Excel file• D2 = HInp
2 – H2
• = 3.8216 – 3.7943 = 0.0273 bit
• Total divergence, D1 + D2 = 0.090 + 0.0273 = 0.1173 bit
Divergence from independence – Conditional Entropy
- Compare different sequences using H to establish relationships
- Given the knowledge of one sequence, say X, can we estimate the uncertainty of Y relative to X ?
- Relation between X, Y, and the conditional entropy, H(X|Y) and H(Y|X)
- conditional entropy is the uncertainty relative to known informationH(X,Y) = H(Y|X) + H(X) = uncertainty of Y given knowledge of X, H(Y|X) + uncertainty of X, sum to the entropy of X and Y = H(X|Y) + H(Y)
Y=2x
log10Y=xlog102X=log10Y/log102
H(Y|X) = H(X,Y) – H(X) = 1.85 – 0.97 = 0.88 bit
11,
n
ji ijp
where
Shannon Entropy – Mutual Information
Joint entropy H(X,Y)
where pij is the joint probability of finding xi and yj
• probability of finding (X,Y) • p00 = 0.1, p01 = 0.3, p10 = 0.4, p11 = 0.2
Mutual information entropy, M(X,Y)• Information shared by X and Y, or it can be us
ed as a similarity measure between X and Y• H(X,Y)= H(X) + H(Y) – M(X,Y) like in se
t theory, A B = A + B – ∪ (A∩B)
• M(X,Y)= H(X) + H(Y) - H(X,Y) • = H(X) – H(X|Y)• = H(Y) – H(Y|X)• = 1.00 – 0.88• M(X,Y)= H(X) + H(Y) – H(X,Y) = 0.97 + 1.00 – 1.85 = 0.12 bit
ji
ijij ppYXH,
log),( 11,
n
ji ijpand
Shannon Entropy – Conditional Entropy
)(0 Xf x
)0|1(log()0|1()0|0(log()0|0( XYfXYfXYfXYf
)|()()|(
))|(log()|()|(
xXYHXfXYH
xXYfxXYfxXYH
xx
y
87549.0
)]6
2log
6
2
6
4log
6
4(
10
6)
4
3log
4
3
4
1log
4
1(
10
4[)|(
XYH
H(Y|X) f(Y|X=x) 0 0 1/4 1 0 3/4 0 1 4/6 1 1 2/6
Conditional entropy
a particular x
All x’s
)(1 Xf x