Springer Texts in Statistics

For further volumes:http://www.springer.com/series/417

Allan Gut

Probability: A GraduateCourse

Second Edition

123

Allan GutDepartment of MathematicsUppsala UniversityUppsalaSweden

ISSN 1431-875XISBN 978-1-4614-4707-8 ISBN 978-1-4614-4708-5 (eBook)DOI 10.1007/978-1-4614-4708-5Springer New York Heidelberg Dordrecht London

Library of Congress Control Number: 2012941281

� Springer Science+Business Media New York 2013This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part ofthe material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformation storage and retrieval, electronic adaptation, computer software, or by similar or dissimilarmethodology now known or hereafter developed. Exempted from this legal reservation are briefexcerpts in connection with reviews or scholarly analysis or material supplied specifically for thepurpose of being entered and executed on a computer system, for exclusive use by the purchaser of thework. Duplication of this publication or parts thereof is permitted only under the provisions ofthe Copyright Law of the Publisher’s location, in its current version, and permission for use must alwaysbe obtained from Springer. Permissions for use may be obtained through RightsLink at the CopyrightClearance Center. Violations are liable to prosecution under the respective Copyright Law.The use of general descriptive names, registered names, trademarks, service marks, etc. in thispublication does not imply, even in the absence of a specific statement, that such names are exemptfrom the relevant protective laws and regulations and therefore free for general use.While the advice and information in this book are believed to be true and accurate at the date ofpublication, neither the authors nor the editors nor the publisher can accept any legal responsibility forany errors or omissions that may be made. The publisher makes no warranty, express or implied, withrespect to the material contained herein.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

Preface to the First Edition

Toss a symmetric coin twice. What is the probability that both tosses will yield ahead?

This is a well-known problem that anyone can solve. Namely, the probability ofa head in each toss is 1=2, so the probability of two consecutive heads is1=2 � 1=2 ¼ 1=4.

BUT! What did we do? What is involved in the solution? What are the argu-ments behind our computations? Why did we multiply the two halves connectedwith each toss?

This is reminiscent of the centipede1 who was asked by another animal how hewalks; he who has so many legs, in which order does he move them as he iswalking? The centipede contemplated the question for a while, but found noanswer. However, from that moment on he could no longer walk.

This book is written with the hope that we are not centipedes.There exist two kinds of probabilists. One of them is the mathematician who

views probability theory as a purely mathematical discipline, like algebra, topol-ogy, differential equations, and so on. The other kind views probability theory asthe mathematical modeling of random phenomena, that is with a view towardapplications, and as a companion to statistics, which aims at finding methods,principles, and criteria in order to analyze data emanating from experimentsinvolving random phenomena and other observations from the real world, with theultimate goal of making wise decisions. I would like to think of myself as both.

What kind of a random process describes the arrival of claims at an insurancecompany? Is it one process or should one rather think of different processes, suchas one for claims concerning stolen bikes and one for houses that have burntdown? How well should the DNA sequences of an accused offender and a piece ofevidence match each other in order for a conviction? A milder version is how toorder different species in a phylogenetic tree. What are the arrival rates of

1 Cent is 100, so it means an animal with 100 legs. In Swedish the name of the animal istusenfoting, where ‘‘tusen’’ means 1,000 and ‘‘fot’’ is foot; thus an animal with 1,000 legs or feet.

v

customers to a grocery store? How long are the service times? How do theclapidemia cells split? Will they create a new epidemic or can we expect them todie out? A classical application has been the arrivals of telephone calls to aswitchboard and the duration of calls. Recent research and model testing con-cerning the Internet traffic has shown that the classical models break down com-pletely and new thinking has become necessary. And, last but (not?) least, there aremany games and lotteries.

The aim of this book is to provide the reader with a fairly thorough treatment ofthe main body of basic and classical probability theory, preceded by an intro-duction to the mathematics which is necessary for a solid treatment of the material.This means that we begin with basics from measure theory, such as �-algebras, settheory, measurability (random variables), and Lebesgue integration (expectation),after which we turn to the Borel–Cantelli lemmas, inequalities, transforms, and thethree classical limit theorems: the law of large numbers, the central limit theorem,and the law of the iterated logarithm. A final chapter on martingales—one of themost efficient, important, and useful tools in probability theory—is preceded by achapter on topics that could have been included with the hope that the reader willbe tempted to look further into the literature. The reason that these topics did notget a chapter of their own is that beyond a certain number of pages a book becomesdeterring rather than tempting (or, as somebody said with respect to an earlier bookof mine: ‘‘It is a nice format for bedside reading’’).

One thing that is not included in this book is a philosophical discussion ofwhether or not chance exists, whether or not randomness exists. On the other hand,probabilistic modeling is a wonderful, realistic, and efficient way to model phe-nomena containing uncertainties and ambiguities, regardless of whether or not theanswer to the philosophical question is yes or no.

I remember having read somewhere a sentence like ‘‘There exist already somany textbooks [of the current kind], so, why do I write another one?’’ Thissentence could equally well serve as an opening for the present book.

Luckily, I can provide an answer to that question. The answer is the shortversion of the story of the mathematician who was asked how one realizes that thefact he presented in his lecture (because this was really a he) was trivial. After2 min of complete silence he mumbled

I know it is trivial, but I have forgotten why.

I strongly dislike the arrogance and snobbism that encompasses mathematics andmany mathematicians. Books and papers are filled with expressions such as ‘‘it iseasily seen’’, ‘‘it is trivial’’, ‘‘routine computations yield’’, and so on. The lastexample is sometimes modified into ‘‘routine, but tedious, computations yield’’.And we all know that behind things that are easily seen there may be years ofthinking and/or huge piles of scrap notes that lead nowhere, and one sheet whereeverything finally worked out nicely.

Clearly, things become routine after many years. Clearly, facts become, at leastintuitively, obvious after some decades. But in writing papers and books we try to

vi Preface to the First Edition

help those who do not know yet, those who want to learn. We wish to attractpeople to this fascinating part of the world. Unfortunately though, phrases like theabove ones are repellent, rather than being attractive. If a reader understandsimmediately that is fine. However, it is more likely that he or she starts off withsomething that either results in a pile of scrap notes or in frustration. Or both. Andnobody is made happier, certainly not the reader. I have therefore avoided, or, atleast, tried to avoid, expressions like the above unless they are adequate.

The main aim of a book is to be helpful to the reader, to help her or him tounderstand, to inform, to educate, and to attract (and not for the author to provehimself to the world). It is therefore essential to keep the flow, not only in thewriting, but also in the reading. In the writing it is therefore of great importance tobe rather extensive and not to leave too much to the (interested) reader.

A related aspect concerns the style of writing. Most textbooks introduce thereader to a number of topics in such a way that further insights are gained throughexercises and problems, some of which are not at all easy to solve, let alone trivial.We take a somewhat different approach in that several such ‘‘would have been’’exercises are given, together with their solutions as part of the ordinary text—which, as a side effect, reduces the number of exercises and problems at the end ofeach chapter. We also provide, at times, results for which the proofs consist ofvariations of earlier ones, and therefore are left as an exercise, with the motivationthat doing almost the same thing as somebody else has done provides a muchbetter understanding than reading, nodding, and agreeing. I also hope that thisapproach creates the atmosphere of a dialog rather than of the more traditionalmonolog (or sermon).

The ultimate dream is, of course, that this book contains no errors, no slips, nomisprints. Henrik Wanntorp has gone over a substantial part of the manuscriptwith a magnifying glass, thereby contributing immensely to making that dreamcome true. My heartfelt thanks, Henrik. I also wish to thank Raimundas Gaigalasfor several perspicacious remarks and suggestions concerning his favorite sections,and a number of reviewers for their helpful comments and valuable advice. Asalways, I owe a lot to Svante Janson for being available for any question at alltimes, and, more particularly, for always providing me with an answer. JohnKimmel of Springer-Verlag has seen me through the process with a uniquecombination of professionalism, efficiency, enthusiasm, and care, for which I ammost grateful.

Finally, my hope is that the reader who has digested this book is ready andcapable to attack any other text, for which a solid probabilistic foundation isnecessary or, at least, desirable.

Uppsala, November 2004 Allan Gut

Preface to the First Edition vii

Preface to the Second Edition

In this, second, edition the original text has been extended a bit, some results havebeen updated, around a dozen references have been added, and a proof that wasupside down has been turned upside down. Some subtractions, such as an incorrecttheorem and a silly problem, are also included. Needless to say, a number of typos,some of which have been sent to me from various parts of the world, some ofwhich I have discovered myself, have been corrected. In addition, I have vacuumcleaned the manuscript, which has resulted in the modification (or deletion) ofsome obscure, incomplete, or otherwise fuzzy arguments or explanations, hopingthat they have become more elucidating or, at least, transparent.

In the preface to the first edition I wrote ‘‘The ultimate dream is, of course, thatthis book contains no errors, no slips, no misprints’’. Given current experience I amafraid that, although closer, we might still not be quite there. Nevertheless, I wishto thank those who have contributed toward this goal.

A special thanks goes to Katja Trinajstic for her careful scrutiny of the book,while taking it for credits, and for her many perspicacious observations andremarks. And, as always, it has been great to have Svante Janson around wheneverdesperation was threatening.

Finally, I wish to thank Marc Strauss at Springer-Verlag for the friendlyenthusiasm that he has transmitted to me during the publication process.

Uppsala, April 2012 Allan Gut

ix

Contents

1 Introductory Measure Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Probability Theory: An Introduction . . . . . . . . . . . . . . . . . . . . 12 Basics from Measure Theory . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Collections of Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4 A Metatheorem and Some Consequences . . . . . . . . . . . . 9

3 The Probability Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1 Limits and Completeness . . . . . . . . . . . . . . . . . . . . . . . 113.2 An Approximation Lemma . . . . . . . . . . . . . . . . . . . . . . 133.3 The Borel Sets on R . . . . . . . . . . . . . . . . . . . . . . . . . . 143.4 The Borel Sets on R

n . . . . . . . . . . . . . . . . . . . . . . . . . . 164 Independence; Conditional Probabilities. . . . . . . . . . . . . . . . . . 16

4.1 The Law of Total Probability; Bayes’ Formula . . . . . . . . 174.2 Independence of Collections of Events . . . . . . . . . . . . . . 184.3 Pair-wise Independence . . . . . . . . . . . . . . . . . . . . . . . . 19

5 The Kolmogorov Zero–One Law . . . . . . . . . . . . . . . . . . . . . . 206 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Definition and Basic Properties. . . . . . . . . . . . . . . . . . . . . . . . 25

1.1 Functions of Random Variables. . . . . . . . . . . . . . . . . . . 272 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.1 Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . 302.2 Integration: A Preview . . . . . . . . . . . . . . . . . . . . . . . . . 322.3 Decomposition of Distributions . . . . . . . . . . . . . . . . . . . 362.4 Some Standard Discrete Distributions . . . . . . . . . . . . . . 392.5 Some Standard Absolutely Continuous

Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

xi

2.6 The Cantor Distribution . . . . . . . . . . . . . . . . . . . . . . . . 402.7 Two Perverse Examples . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Random Vectors; Random Elements . . . . . . . . . . . . . . . . . . . . 423.1 Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.2 Random Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4 Expectation; Definitions and Basics. . . . . . . . . . . . . . . . . . . . . 454.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.2 Basic Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5 Expectation; Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 546 Indefinite Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 A Change of Variables Formula . . . . . . . . . . . . . . . . . . . . . . . 608 Moments, Mean, Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . 629 Product Spaces; Fubini’s Theorem . . . . . . . . . . . . . . . . . . . . . 64

9.1 Finite-Dimensional Product Measures . . . . . . . . . . . . . . 649.2 Fubini’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 659.3 Partial Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 659.4 The Convolution Formula . . . . . . . . . . . . . . . . . . . . . . . 67

10 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6810.1 Independence of Functions of Random

Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7110.2 Independence of �-Algebras . . . . . . . . . . . . . . . . . . . . . 7110.3 Pair-Wise Independence . . . . . . . . . . . . . . . . . . . . . . . . 7110.4 The Kolmogorov Zero–One Law Revisited. . . . . . . . . . . 72

11 The Cantor Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7312 Tail Probabilities and Moments . . . . . . . . . . . . . . . . . . . . . . . 7413 Conditional Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7914 Distributions with Random Parameters . . . . . . . . . . . . . . . . . . 8115 Sums of a Random Number of Random Variables . . . . . . . . . . 83

15.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8516 Random Walks; Renewal Theory . . . . . . . . . . . . . . . . . . . . . . 88

16.1 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8816.2 Renewal Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8916.3 Renewal Theory for Random Walks . . . . . . . . . . . . . . . 9016.4 The Likelihood Ratio Test . . . . . . . . . . . . . . . . . . . . . . 9116.5 Sequential Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 9116.6 Replacement Based on Age . . . . . . . . . . . . . . . . . . . . . 92

17 Extremes; Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9317.1 Extremes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9317.2 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

18 Borel–Cantelli Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9618.1 The Borel–Cantelli Lemmas 1 and 2 . . . . . . . . . . . . . . . 9618.2 Some (Very) Elementary Examples . . . . . . . . . . . . . . . . 9818.3 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10118.4 Recurrence and Transience of Simple

Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

xii Contents

18.5P1

n¼1 PðAnÞ ¼ 1 and PðAn i:o:Þ ¼ 0 . . . . . . . . . . . . . . . 10418.6 Pair-Wise Independence . . . . . . . . . . . . . . . . . . . . . . . . 10418.7 Generalizations Without Independence . . . . . . . . . . . . . . 10518.8 Extremes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10618.9 Further Generalizations . . . . . . . . . . . . . . . . . . . . . . . . 109

19 A Convolution Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11320 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

3 Inequalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1191 Tail Probabilities Estimated Via Moments . . . . . . . . . . . . . . . . 1192 Moment Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1273 Covariance: Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1304 Interlude on Lp-Spaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1315 Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1326 Symmetrization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1337 Probability Inequalities for Maxima. . . . . . . . . . . . . . . . . . . . . 1388 The Marcinkiewics–Zygmund Inequalities . . . . . . . . . . . . . . . . 1469 Rosenthal’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15110 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

4 Characteristic Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1571 Definition and Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

1.1 Uniqueness; Inversion . . . . . . . . . . . . . . . . . . . . . . . . . 1591.2 Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1641.3 Some Further Results . . . . . . . . . . . . . . . . . . . . . . . . . . 165

2 Some Special Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1662.1 The Cantor Distribution . . . . . . . . . . . . . . . . . . . . . . . . 1662.2 The Convolution Table Revisited . . . . . . . . . . . . . . . . . 1682.3 The Cauchy Distribution. . . . . . . . . . . . . . . . . . . . . . . . 1702.4 Symmetric Stable Distributions . . . . . . . . . . . . . . . . . . . 1712.5 Parseval’s Relation . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

3 Two Surprises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1734 Refinements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1755 Characteristic Functions of Random Vectors . . . . . . . . . . . . . . 180

5.1 The Multivariate Normal Distribution . . . . . . . . . . . . . . 1805.2 The Mean and the Sample Variance

Are Independent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1836 The Cumulant Generating Function . . . . . . . . . . . . . . . . . . . . . 1847 The Probability Generating Function . . . . . . . . . . . . . . . . . . . . 186

7.1 Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1888 The Moment Generating Function. . . . . . . . . . . . . . . . . . . . . . 189

8.1 Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1918.2 Two Boundary Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 191

9 Sums of a Random Number of Random Variables . . . . . . . . . . 192

Contents xiii

10 The Moment Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19410.1 The Moment Problem for Random Sums . . . . . . . . . . . . 196

11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

5 Convergence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2011 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

1.1 Continuity Points and Continuity Sets . . . . . . . . . . . . . . 2031.2 Measurability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2051.3 Some Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

2 Uniqueness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2073 Relations Between Convergence Concepts . . . . . . . . . . . . . . . . 208

3.1 Converses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2114 Uniform Integrability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2135 Convergence of Moments. . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

5.1 Almost Sure Convergence . . . . . . . . . . . . . . . . . . . . . . 2185.2 Convergence in Probability . . . . . . . . . . . . . . . . . . . . . . 2195.3 Convergence in Distribution . . . . . . . . . . . . . . . . . . . . . 222

6 Distributional Convergence Revisited . . . . . . . . . . . . . . . . . . . 2246.1 Scheffé’s Lemma. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

7 A Subsequence Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2288 Vague Convergence; Helly’s Theorem. . . . . . . . . . . . . . . . . . . 230

8.1 Vague Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 2308.2 Helly’s Selection Principle . . . . . . . . . . . . . . . . . . . . . . 2318.3 Vague Convergence and Tightness . . . . . . . . . . . . . . . . 2348.4 The Method of Moments . . . . . . . . . . . . . . . . . . . . . . . 236

9 Continuity Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2379.1 The Characteristic Function . . . . . . . . . . . . . . . . . . . . . 2379.2 The Cumulant Generating Function . . . . . . . . . . . . . . . . 2409.3 The (Probability) Generating Function . . . . . . . . . . . . . . 2409.4 The Moment Generating Function . . . . . . . . . . . . . . . . . 241

10 Convergence of Functions of Random Variables . . . . . . . . . . . . 24310.1 The Continuous Mapping Theorem . . . . . . . . . . . . . . . . 244

11 Convergence of Sums of Sequences . . . . . . . . . . . . . . . . . . . . 24611.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24811.2 Converses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25111.3 Symmetrization and Desymmetrization. . . . . . . . . . . . . . 254

12 Cauchy Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25513 Skorohod’s Representation Theorem . . . . . . . . . . . . . . . . . . . . 25714 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

6 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2651 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

1.1 Convergence Equivalence . . . . . . . . . . . . . . . . . . . . . . . 2661.2 Distributional Equivalence . . . . . . . . . . . . . . . . . . . . . . 267

xiv Contents

1.3 Sums and Maxima . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2681.4 Moments and Tails . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

2 A Weak Law for Partial Maxima . . . . . . . . . . . . . . . . . . . . . . 2693 The Weak Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . 270

3.1 Two Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2764 A Weak Law Without Finite Mean . . . . . . . . . . . . . . . . . . . . . 278

4.1 The St. Petersburg Game . . . . . . . . . . . . . . . . . . . . . . . 2835 Convergence of Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

5.1 The Kolmogorov Convergence Criterion . . . . . . . . . . . . 2865.2 A Preliminary Strong Law . . . . . . . . . . . . . . . . . . . . . . 2885.3 The Kolmogorov Three-Series Theorem . . . . . . . . . . . . . 2895.4 Lévy’s Theorem on the Convergence of Series . . . . . . . . 292

6 The Strong Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . 2947 The Marcinkiewicz–Zygmund Strong Law . . . . . . . . . . . . . . . . 2988 Randomly Indexed Sequences. . . . . . . . . . . . . . . . . . . . . . . . . 3019 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

9.1 Normal Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3059.2 The Glivenko–Cantelli Theorem . . . . . . . . . . . . . . . . . . 3069.3 Renewal Theory for Random Walks . . . . . . . . . . . . . . . 3069.4 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

10 Uniform Integrability; Moment Convergence . . . . . . . . . . . . . . 30911 Complete Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

11.1 The Hsu–Robbins–Erd}os Strong Law. . . . . . . . . . . . . . . 31211.2 Complete Convergence and the Strong Law . . . . . . . . . . 314

12 Some Additional Results and Remarks . . . . . . . . . . . . . . . . . . 31512.1 Convergence Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . 31512.2 Counting Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 32012.3 The Case r ¼ p Revisited . . . . . . . . . . . . . . . . . . . . . . . 32112.4 Random Indices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32212.5 Delayed Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

13 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

7 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3291 The i.i.d. Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3302 The Lindeberg–Lévy–Feller Theorem . . . . . . . . . . . . . . . . . . . 330

2.1 Lyapounov’s Condition . . . . . . . . . . . . . . . . . . . . . . . . 3392.2 Remarks and Complements. . . . . . . . . . . . . . . . . . . . . . 3402.3 Pair-Wise Independence . . . . . . . . . . . . . . . . . . . . . . . . 3432.4 The Central Limit Theorem for Arrays. . . . . . . . . . . . . . 344

3 Anscombe’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3454 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

4.1 The Delta Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3494.2 Stirling’s Formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3504.3 Renewal Theory for Random Walks . . . . . . . . . . . . . . . 351

Contents xv

4.4 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3525 Uniform Integrability; Moment Convergence . . . . . . . . . . . . . . 3536 Remainder Term Estimates. . . . . . . . . . . . . . . . . . . . . . . . . . . 355

6.1 The Berry–Esseen Theorem . . . . . . . . . . . . . . . . . . . . . 3556.2 Proof of the Berry–Esseen Theorem 6.2 . . . . . . . . . . . . . 357

7 Some Additional Results and Remarks . . . . . . . . . . . . . . . . . . 3637.1 Rates of Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3637.2 Non-Uniform Estimates . . . . . . . . . . . . . . . . . . . . . . . . 3647.3 Renewal Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3647.4 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3657.5 Local Limit Theorems . . . . . . . . . . . . . . . . . . . . . . . . . 3657.6 Large Deviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3667.7 Convergence Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . 3677.8 Precise Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3727.9 A Short Outlook on Extensions . . . . . . . . . . . . . . . . . . . 375

8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

8 The Law of the Iterated Logarithm. . . . . . . . . . . . . . . . . . . . . . . 3831 The Kolmogorov and Hartman–Wintner LILs. . . . . . . . . . . . . . 384

1.1 Outline of Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3852 Exponential Bounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3853 Proof of the Hartman–Wintner Theorem . . . . . . . . . . . . . . . . . 3874 Proof of the Converse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3965 The LIL for Subsequences . . . . . . . . . . . . . . . . . . . . . . . . . . . 398

5.1 A Borel–Cantelli Sum for Subsequences . . . . . . . . . . . . 4015.2 Proof of Theorem 5.2. . . . . . . . . . . . . . . . . . . . . . . . . . 4025.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404

6 Cluster Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4046.1 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406

7 Some Additional Results and Remarks . . . . . . . . . . . . . . . . . . 4127.1 Hartman–Wintner via Berry–Esseen. . . . . . . . . . . . . . . . 4127.2 Examples Not Covered by

Theorems 5.2 and 5.1. . . . . . . . . . . . . . . . . . . . . . . . . . 4137.3 Further Remarks on Sparse Subsequences. . . . . . . . . . . . 4147.4 An Anscombe LIL. . . . . . . . . . . . . . . . . . . . . . . . . . . . 4167.5 Renewal Theory for Random Walks . . . . . . . . . . . . . . . 4177.6 Record Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4177.7 Convergence Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . 4187.8 Precise Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . 4197.9 The Other LIL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4197.10 Delayed Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420

8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421

xvi Contents

9 Limit Theorems: Extensions and Generalizations. . . . . . . . . . . . . 4231 Stable Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4242 The Convergence to Types Theorem . . . . . . . . . . . . . . . . . . . . 4273 Domains of Attraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430

3.1 Sketch of Preliminary Steps . . . . . . . . . . . . . . . . . . . . . 4333.2 Proof of Theorems 3.2 and 3.3 . . . . . . . . . . . . . . . . . . . 4353.3 Two Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4383.4 Two Variations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4393.5 Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440

4 Infinitely Divisible Distributions . . . . . . . . . . . . . . . . . . . . . . . 4425 Sums of Dependent Random Variables . . . . . . . . . . . . . . . . . . 4486 Convergence of Extremes . . . . . . . . . . . . . . . . . . . . . . . . . . . 451

6.1 Max-Stable and Extremal Distributions . . . . . . . . . . . . . 4516.2 Domains of Attraction . . . . . . . . . . . . . . . . . . . . . . . . . 4566.3 Record Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457

7 The Stein–Chen Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4598 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464

10 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4671 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468

1.1 Properties of Conditional Expectation . . . . . . . . . . . . . . 4711.2 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4741.3 The Rao–Blackwell Theorem . . . . . . . . . . . . . . . . . . . . 4751.4 Conditional Moment Inequalities . . . . . . . . . . . . . . . . . . 476

2 Martingale Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4772.1 The Defining Relation . . . . . . . . . . . . . . . . . . . . . . . . . 4792.2 Two Equivalent Definitions . . . . . . . . . . . . . . . . . . . . . 479

3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4814 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4875 Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4886 Stopping Times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4917 Doob’s Optional Sampling Theorem . . . . . . . . . . . . . . . . . . . . 4958 Joining and Stopping Martingales . . . . . . . . . . . . . . . . . . . . . . 4979 Martingale Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50110 Convergence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508

10.1 Garsia’s Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50810.2 The Upcrossings Proof . . . . . . . . . . . . . . . . . . . . . . . . . 51110.3 Some Remarks on Additional Proofs . . . . . . . . . . . . . . . 51410.4 Some Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51510.5 A Non-Convergent Martingale . . . . . . . . . . . . . . . . . . . 51510.6 A Central Limit Theorem? . . . . . . . . . . . . . . . . . . . . . . 515

11 The Martingale EðZ j F nÞ . . . . . . . . . . . . . . . . . . . . . . . . . . . 51612 Regular Martingales and Submartingales . . . . . . . . . . . . . . . . . 517

12.1 A Main Martingale Theorem. . . . . . . . . . . . . . . . . . . . . 517

Contents xvii

12.2 A Main Submartingale Theorem . . . . . . . . . . . . . . . . . . 51812.3 Two Non-regular Martingales . . . . . . . . . . . . . . . . . . . . 51912.4 Regular Martingales Revisited. . . . . . . . . . . . . . . . . . . . 520

13 The Kolmogorov Zero–One Law . . . . . . . . . . . . . . . . . . . . . . 52014 Stopped Random Walks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521

14.1 Finiteness of Moments . . . . . . . . . . . . . . . . . . . . . . . . . 52114.2 The Wald Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 52314.3 Tossing a Coin Until Success . . . . . . . . . . . . . . . . . . . . 52514.4 The Gambler’s Ruin Problem . . . . . . . . . . . . . . . . . . . . 52614.5 A Converse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530

15 Regularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53215.1 First Passage Times for Random Walks . . . . . . . . . . . . . 53515.2 Complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53715.3 The Wald Fundamental Identity . . . . . . . . . . . . . . . . . . 538

16 Reversed Martingales and Submartingales . . . . . . . . . . . . . . . . 54116.1 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . 54516.2 U-Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547

17 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549

Appendix: Some Useful Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . 555

A.1 Taylor Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555A.2 Mill’s Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558A.3 Sums and Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559A.4 Sums and Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560A.5 Convexity; Clarkson’s Inequality . . . . . . . . . . . . . . . . . . . . . . 561A.6 Convergence of (Weighted) Averages . . . . . . . . . . . . . . . . . . . 564A.7 Regularly and Slowly Varying Functions . . . . . . . . . . . . . . . . . 566A.8 Cauchy’s Functional Equation . . . . . . . . . . . . . . . . . . . . . . . . 569A.9 Functions and Dense Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 571

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587

xviii Contents

Outline of Contents

In this extended list of contents, we provide a short expansion of the headings intoa quick overview of the contents of the book.

Chapter 1. Introductory Measure TheoryThe mathematical foundation of probability theory is measure theory and thetheory of Lebesgue integration. The bulk of the introductory chapter is devoted tomeasure theory: sets, measurability, �-algebras, and so on. We do not aim at a fullcourse in measure theory, rather to provide enough background for a solid treat-ment of what follows.

Chapter 2. Random VariablesHaving set the scene, the first thing to do is to forget probability spaces (!). Moreprecisely, for modeling random experiments one is interested in certain specificquantities, called random variables, rather than in the underlying probability spaceitself. In Chap. 2 we introduce random variables and present the basic concepts, aswell as concrete applications and examples of probability models. In particular,Lebesgue integration is developed in terms of expectation of random variables.

Chapter 3. InequalitiesSome of the most useful tools in probability theory and mathematics for provingfiniteness or convergence of sums and integrals are inequalities. There exist manyuseful ones spread out in books and papers. In Chap. 3 we make an attempt topresent a sizable amount of the most important inequalities.

Chapter 4. Characteristic FunctionsJust as there are i.a. Fourier transforms that transform convolution of functions intomultiplication of their corresponding transforms, there exist probabilistic trans-forms that ‘‘map’’ addition of independent random variables into multiplication oftheir transforms, the most prominent one being the characteristic function.

xix

Chapter 5. ConvergenceOnce we know how to add random variables the natural problem is to investigateasymptotics. We begin by introducing some convergence concepts, proveuniqueness, after which we investigate how and when they imply each other. Otherimportant problems are when, and to what extent, limits and expectation (limitsand integrals) can be interchanged, and when, and to what extent, functions ofconvergent sequences converge to the function of the limit.

Chapter 6. The Law of Large NumbersThe law of large numbers states that (the distribution of) the arithmetic mean of asequence of independent trials stabilizes around the center of gravity of theunderlying distribution (under suitable conditions). There exist weak and stronglaws and several variations and extensions. We shall meet some of them as well assome applications.

Chapter 7. The Central Limit TheoremThe central limit theorem, which (in its simplest form) states that if the variance isfinite then the arithmetic mean, properly rescaled, of a sequence of independent trialsapproaches a normal distribution as the number of observations increases. Thereexist many variations and generalizations of the theorem, the central one being theLindeberg–Lévy–Feller theorem. We also prove results on moment convergenceand rate results, the foremost one being the celebrated Berry–Esseen theorem.

Chapter 8. The Law of the Iterated LogarithmThis is a special, rather delicate and technical, and very beautiful result, whichprovides precise bounds on the oscillations of sums of the above kind. The nameobviously stems from the iterated logarithm that appears in the expression of theparabolic bound.

Chapter 9. Limit Theorems: Extensions and GeneralizationsThere are a number of additional topics that would fit well into a text like the presentone, but for which there is no room. In this chapter, we shall meet a number ofthem—stable distributions, domains of attraction, infinite divisibility, sums ofdependent random variables, extreme value theory, the Stein–Chen method—in asomewhat more sketchy or introductory style. The reader who gets hooked on such atopic will be advised to some relevant literature (more can be found via the Internet).

Chapter 10. MartingalesThis final chapter is devoted to one of the most central topics, not only in probabilitytheory, but also in more traditional mathematics. Following some introductorymaterial on conditional expectation and the definition of a martingale, we presentseveral examples, convergence results, results for stopped martingales, regularmartingales, uniformly integrable martingales, stopped random walks, and reversedmartingales.

xx Outline of Contents

In AdditionA list of notation and symbols precedes the main body of text, and an appendix withsome mathematical tools and facts, a bibliography, and an index conclude the book.References are provided for more recent results, for more nontraditional material,and to some of the historic sources, but in general not to the more traditionalmaterial. In addition to cited material, the list of references contains references topapers and books that are relevant without having been specifically cited.

Suggestions for a Course CurriculumOne aim with the book is that it should serve as a graduate probability course—as thetitle suggests. In the same way as the sections in Chap. 9 contain materials that nodoubt would have deserved chapters of their own, Chaps. 6–8 contain sectionsentitled ‘‘Some Additional Results and Remarks’’, in which a number of additionalresults and remarks are presented, results that are not as central and basic as earlierones in these chapters.

An adequate course would, in my opinion, consist of Chaps. 1–8, and 10, exceptfor the sections ‘‘Some Additional Results and Remarks’’, plus a skimming throughChap. 9 at the level of the instructor’s preferences.

Outline of Contents xxi

Notation and Symbols

› The sample space! An elementary eventF The �-algebra of eventsxþ maxfx; 0gx� �minfx; 0g½x� The integer part of xlogþ x maxf1; log xg� The ratio of the quantities on either side tends to 1N The (positive) natural numbersZ The integersR The real numbersR The Borel �-algebra on R

‚ð�Þ Lebesgue measurekxkp The Lp-norm of x

Q The rational numbersC The complex numbersC The continuous functionsC0 The functions in C tending to 0 at �1CB The bounded continuous functionsC½a; b� The functions in C with support on the interval ½a; b�D The right-continuous, functions with left-hand limitsD½a; b� The functions in D with support on the interval ½a; b�Dþ The non-decreasing functions in DJG The discontinuities of G 2 DIfAg Indicator function of (the set) A#fAg Number of elements in (cardinality of) AjAj Number of elements in (cardinality of) AAc Complement of A@A The boundary of APðAÞ Probability of A

xxiii

X; Y; Z; . . . Random variablesFðxÞ; FXðxÞ Distribution function (of X)X 2 F X has distribution (function) FCðFXÞ The continuity set of FX

pðxÞ; pXðxÞ Probability function (of X)f ðxÞ; fXðxÞ Density (function) (of X)X; Y; Z; . . . Random (column) vectorsX0; Y0; Z0; . . . The transpose of the vectorsFX;Yðx; yÞ Joint distribution function (of X and Y)pX;Yðx; yÞ Joint probability function (of X and Y)fX;Yðx; yÞ Joint density (function) (of X and Y)E; E X Expectation (mean), expected value of XVar; Var X Variance, variance of XCov ðX; YÞ Covariance of X and Y‰; ‰X;Y Correlation coefficient (between X and Y)

kXkp ¼ ðEjXjpÞ1=p The Lp-norm of X

med ðXÞ Median of XgðtÞ; gXðtÞ (Probability) generating function (of X)ˆðtÞ; ˆXðtÞ Moment generating function (of X)’ðtÞ; ’XðtÞ Characteristic function (of X)X� Y X and Y are equivalent random variables

X¼a:s:Y X and Y are equal (point-wise) almost surely

X¼d Y X and Y are equidistributed

Xn!a:s:

X Xn converges almost surely (a.s.) to X

Xn!p

X Xn converges in probability to X

Xn!r

X Xn converges in r-mean (Lr) to X

Xn!d

X Xn converges in distribution to X

Xn 6!a:s: Xn does not converge almost surely

Xn 6!p Xn does not converge in probability

Xn 6!r Xn does not converge in r-mean (Lr)

Xn 6!d Xn does not converge in distribution

'ðxÞ Standard normal distribution function`ðxÞ Standard normal density (function)F 2 DðGÞ F belongs to the domain of attraction of Gg 2 RV ð‰Þ g varies regularly at infinity with exponent ‰g 2 SV g varies slowly at infinityBeðpÞ Bernoulli distributionflðr; sÞ Beta distributionBinðn; pÞ Binomial distributionCðm; aÞ Cauchy distribution

xxiv Notation and Symbols

´2ðnÞ Chi-square distribution–ðaÞ One-point distributionExpðaÞ Exponential distributionFðm; nÞ (Fisher’s) F-distributionFsðpÞ First success distribution¡ðp; aÞ Gamma distributionGeðpÞ Geometric distributionHðN; n; pÞ Hypergeometric distributionLðaÞ Laplace distributionLNð„;�2Þ Log-normal distributionNð„;�2Þ Normal distributionNð0; 1Þ Standard normal distributionNBinðn; pÞ Negative binomial distributionPaðk;fiÞ Pareto distributionPoðmÞ Poisson distributionRaðfiÞ Rayleigh distributiontðnÞ (Student’s) t-distributionTriða; bÞ Triangular distribution on ða; bÞUða; bÞ Uniform or rectangular distribution on ða; bÞWða; bÞ Weibull distributionX 2 Pð�Þ X has a P-distribution with parameter �X 2 Pðfi;flÞ X has a P-distribution with parameters fi and fla.e. Almost everywherea.s. Almost surelycf. Confer, compare, take counseli.a. Inter alia, among other things, such asi.e. Id est, that isi.o. Infinitely ofteniff If and only ifi.i.d. Independent, identically distributedviz. Videlicet, in whichw.l.o.g. Without loss of generality€ Hint for solving a problem| Bonus remark in connection with a problemh End of proof, definitions, exercises, remarks, etc.

Notation and Symbols xxv