Statistics for Socialand Behavioral Sciences

Session #18:Literary Analysis using Tests

(Agresti and Finlay, from Chapter 5 to Chapter 6)

Prof. Amine Ouazad

Outline

1. True or False?

2. The p-value

3. One sided t tests

Next time: one-sided t test of mean and proportion Chapter 6 of A&F

The Federalist Papers• End of 18th century: end of British colonial rule in the 13 colonies.• How should the 13 colonies be ruled?

• Anti-Federalist papers written by unknown authors Brutus and Cato.

• Federalist papers published between October 1787 and August 1788 under the names “Philo-Publius”.

• Promoting the ratification of the United States Constitution,and describing the virtues of the checks and balances, an executive, and the judicial review of laws by federal courts.

• United States Constitution was ratified in 1788.

Source: Inference in an authorship problem: A comparative Study of Discrimination Methods Applied to the Authorship of the Disputed Federalist Papers, Journal of the American Statistical Association, Volume 58, issue 302, 1963.

Exercise 6.20: Literary Analysis

The authorship of an old document is in doubt. A historian hypothesizes that the author was a journalist named Jacalyn Levine. Upon a thorough investigation of Levine’s known works, it is observed that one unusual feature of her writing was that she consistently began 6% of her sentences with the word whereas. To test the historian’s hypothesis, it is decided to count the number of sentences in the disputed document that begin with whereas. Out of the 300 sentences, none do.

Let π denote the probability that any one sentence written by the unknown author of the document begins with whereas.

Test H0: “π= 0.06” against Ha: “π is not equal 0.06.”

What assumptions are needed for your conclusion to be valid?

To know:Hypothesis testing is the foundation of (social) sciences.• Three typical types of hypothesis:

– A parameter (e.g. m) is equal to v.– A parameter (e.g. m) is greater than v.– A parameter (e.g. m) is lower than v.

• Null hypothesis (to be rejected), and alternative hypothesis.• We provide evidence to reject a null hypothesis.

– We might not have evidence to reject the null hypothesis.

For a test on the population mean m: H0 : m = v. Ha: m ≠ v.• Confidence interval method with CI = [m-t*SE, m+t*SE].

– Reject the H0 with significance level 5% if the 95% confidence interval for the sample mean m does not include v.

– Reject the H0 with significance level 10% if the 90% confidence interval for the sample mean m does not include v.

• t-test method with t = (m-v)/SE: – reject the null at 1% if is outside [-t0.005,t0.005].

– reject the null at 5% if is outside [-t0.025,t0.025].

– reject the null at 10% if is outside [-t0.05,t0.05].– With degrees of freedom df = N-1.

t Table

From the confidence interval method …to the t-test

Null hypothesis: m = v.• We do not reject the null hypothesis H0 with

confidence level 95% if the 95% confidence interval for the sample mean m includes v.

Do not reject H0 at 95% if: m – t0.025 * SE < v < m + t0.025 * SE

• Notice that this is equivalent to:Do not reject H0 if:

-t0.025 < (m-v)/SE < t0.025

• t0.025 is the 95% critical value for the t statistic.• (m-v)/SE is the t statistic.

Graphically…

Under the null hypothesis (m=v):• (m-v)/SE follows a standard normal distribution if the sample size is large.• (m-v)/SE follows a t distribution

if (i) the sample size is small and (ii) X is normally distributed.

Sampling distribution of the t statisticdf = N-1

On this graph, indicate for which values of t we should reject the null hypothesis…

• With 95% confidence.• With 90% confidence.

And also with 99% confidence ?

The p-value

is the probability (in other potential samples, unobserved) that the absolute value of the t statistic is greater than the observed t statistic in our sample.• A low p value means…..– that larger absolute values of the t test are unlikely.

In practice:• Reject the null hypothesis H0 at 95% if the p-value

is lower than 0.05.• Similarly for 90% and 99%! (0.10 and 0.01 resp.)

Source: Mosteller, Wallace, Inference in an authorship problem: A comparative Study of Discrimination Methods Applied to the Authorship of the Disputed Federalist Papers, Journal of the American Statistical Association, Volume 58, issue 302, 1963.

Frederick Mosteller,

Harvard University

t statistics

What can we infer? At 95% ? At 90%? At 99%?

• The authors build a measure y of the style of a paper.• Upon, whilst, there, on, etc.

• When Hamilton writes, such measure is on average wH.

• Null hypothesis for paper k: H0 : style = wH.

And similarly for Madison.

Publication bias • Papers get published if the t statistic is

above 1.96 (df = infinite, N is large).– This is a convention in science and social sciences.

• This is a bad thing !!! Non rejected H0 should be reported !!!• Distribution of t statistics across papers:

• Brodeur, Le, Sangnier, Zylberberg, “Star Wars: The Empirics Strike Back”, March 2013.

Number of papers

t statistic

1.96

Outline

1. Literary analysis using hypothesis testing

2. One sided hypothesis and t tests.

Next time: one-sided t test of mean and proportion Chapter 6 of A&F

Hypothesis testing• Hypothesis: an empirical statement about a population

parameter. Usually of the shape:– “The parameter is equal to a given value”– “The parameter is greater than a given value”– “The parameter is lower than a given value”

• Almost all scientific/sociological/economic statements can be reduced to one of these three types.– “The population proportion of voters for Cory Gardner is

greater than 50%.” (second type of hypothesis)– “The impact of ZMapp on Ebola patients’ condition is zero.”

(first type of hypothesis)

Last session

This session

We haven’t seen that yet!

One sided test of H0: m<v, Ha: m>=v

• When testing m<v, use the t statistic method.– avoid the confidence interval method.

• Assume that m=v and build t = (m-v)/SE.• Reject the null hypothesis at 95%

if the t statistic is greater than t0.05.

• Similarly for 90% and 99% (t0.10 and t0.01).

Back to Cory Gardner• With 1802 respondents, the polling company PPP

found that 48% of respondents would vote for Gardner.

• Pollsters think that Gardner wins if the vote share is > 46%.

• Can you test the null hypothesis that the true vote share is <= 46%, with Ha: vote share > 46% ?

Wrap upConfidence interval method for the test of H0 : m = v. Ha: m ≠ v.

– Reject the H0 with significance level 1% if the 99% confidence interval for the sample mean m does not include v.

– Reject the H0 with significance level 5% if the 95% confidence interval for the sample mean m does not include v.

– Reject the H0 with significance level 10% if the 90% confidence interval for the sample mean m does not include v.

t test method for the test of H0 : m = v. Ha: m ≠ v.– Build the t statistic (m-v)/SE– Reject the H0 with significance level 1%

if the t statistic is outside the range [-t0.005 , t0.005] – Reject the H0 with significance level 5%

if the t statistic is outside the range [-t0.025 , t0.025] – Reject the H0 with significance level 10%

if the t statistic is outside the range [-t0.05 , t0.05]

I will not ask for the one sided t-test, but good to know for your future life.

Coming up: Readings:• Mid term on Tuesday, November 25.

– Coverage: up to Chapter 6 inclusive.

• No online quiz this week.• Make sure you come to sessions and recitations.

For help:

• Amine OuazadOffice 1135, Social Science buildingamine.ouazad@nyu.eduOffice hour: Tuesday from 5 to 6.30pm.

• GAF: Irene PanedaIrene.paneda@nyu.eduSunday recitations. At the Academic Resource Center, Monday from 2 to 4pm.