1. Asymptotic Theory

Chapter theme: asymptotic (large-sample) theory. Finite-sample properties require strong assumptions on the distribution \(P\), so we instead study large-sample properties as \(n\to\infty\). §1.1 Three prototypical problems: estimation, testing, confidence regions. §1.2 Convergence in probability (Markov's inequality → weak law of large numbers, WLLN). §1.3 Existence of moments (Jensen's inequality; higher-order moment exists ⇒ lower-order exists; marginal convergence in probability ⇒ joint convergence; the Continuous Mapping Theorem, CMT; consistency vs unbiasedness; empirical distribution & Glivenko-Cantelli). §1.4 Convergence in moments (\(L^q\) convergence ⇒ convergence in probability, but not conversely). §1.5 Convergence in distribution (Portmanteau lemma's 7 equivalent characterizations; its relation to convergence in probability; Slutsky's lemma; the distributional CMT). §1.6 Comparison of convergence notions (almost sure, point-wise; the ordering p.w.⇒a.s.⇒p⇒d, and \(L^q\)⇒p). §1.7 Central limit theorem (univariate + Cramér-Wold + multivariate CLT). §1.8 Hypothesis testing (two types of error; consistency in level; \(p\)-value; confidence region; multidimensional testing via \(\chi^2\); the delta method; correlation & Cauchy-Schwarz; the limiting distribution of the median via Berry-Esseen). §1.9 Tightness (\(\to_d\) ⇒ tight; Prokhorov's theorem; \(\tau_n\)-consistency). §1.10 Stochastic order (\(o_P(1)\), \(O_P(1)\) and their rules of calculus).

Definition 1.1 (Convergence in Probability) A sequence of random vectors \(\{X_n\in\mathbb R^k:n\ge1\}\) converges in probability to a random variable \(X\in\mathbb R^k\) (denoted \(X_n\xrightarrow{p}X\)) if for all \(\varepsilon>0\), $$\mathbb P(|X_n-X|>\varepsilon)\to0\quad\text{as }n\to\infty$$ where \(|X_n-X|\) is the Euclidean distance. Two equivalent statements: (i) \(\lim_{n\to\infty}\mathbb P(|X_n-X|>\varepsilon)=0\); (ii) for any \(\delta,\varepsilon>0\), \(\exists N\) with \(\mathbb P(|X_n-X|>\varepsilon)\le\delta\) for all \(n\ge N\).

Theorem 1.1 (Markov's inequality) For any random variable (vector) \(X\), \(\forall q>0\), \(\forall\varepsilon>0\), $$\mathbb P(|X|>\varepsilon)\le\frac{\mathbb E[|X|^q]}{\varepsilon^q}$$ where \(|\cdot|\) is the Euclidean norm.

Proof (Theorem 1.1) Use "\(g(x)\le f(x)\Rightarrow\mathbb E[g(x)]\le\mathbb E[f(x)]\)". By the definition of the indicator function, $$\mathbf 1\{|X|>\varepsilon\}\le\frac{|X|^q}{\varepsilon^q} \tag{1.1}$$ (when \(|X|>\varepsilon\) the LHS $=1$ and the RHS $>1$; when \(|X|\le\varepsilon\) the LHS $=0$ and the RHS \(\ge0\), so it always holds.) Taking expectations on both sides: $$\mathbb E[\mathbf 1\{|X|>\varepsilon\}]=\mathbb P(|X|>\varepsilon)\le\frac{\mathbb E[|X|^q]}{\varepsilon^q}\quad\blacksquare$$

Definition 1.2 (i.i.d.) \(\{X_n:n\ge1\}\) are independently distributed if \(\mathbb P(X_1\le x_1,\dots,X_n\le x_n)=\prod_{i=1}^n\mathbb P(X_i\le x_i)\); identically distributed if \(\mathbb P(X_i\le x)=\mathbb P(X_j\le x),\forall i,j\). Satisfying both is i.i.d.

Theorem 1.2 (Weak Law of Large Numbers, WLLN) Let \(\{X_n:n\ge1\}\) be an i.i.d. sequence on \(\mathbb R\) with c.d.f. \(P\), and suppose \(\mu(P)\) exists (\(\mathbb E[|\mu(P)|]<\infty\)). Then $$\bar X_n\xrightarrow{p}\mu(P)$$ where the sample mean \(\bar X_n=\frac1n\sum_{i=1}^nX_i\). (The strong law is \(\bar X_n\xrightarrow{a.s.}\mu\).)

Proof (Theorem 1.2, assuming additionally \(\sigma^2(P)<\infty\)) Further assume \(\sigma^2(P)<\infty\). By Markov's inequality (\(q=2\)): $$\mathbb P(|\bar X_n-\mu(P)|>\varepsilon)\le\frac{\mathbb E[(\bar X_n-\mu(P))^2]}{\varepsilon^2}=\frac{\mathrm{Var}(\bar X_n)}{\varepsilon^2}=\frac{1}{n\varepsilon^2}\mathrm{Var}(X_i)\to0 \tag{1.2}$$ so \(\bar X_n\xrightarrow{p}\mu(P)\). \(\blacksquare\)

Definition 1.3 (Existence of the mean) The mean \(\mathbb E[X]\) of a random vector \(X\) exists if \(\mathbb E[|X|]<\infty\). For a random variable on \(\mathbb R\), decompose into positive/negative parts \(X^+\equiv\max\{X,0\}\), \(X^-\equiv\max\{-X,0\}\); then \(\mathbb E[X]=\mathbb E[X^+]-\mathbb E[X^-]\) and \(\mathbb E[|X|]=\mathbb E[X^+]+\mathbb E[X^-]\). So \(\mathbb E[|X|]<\infty\iff\mathbb E[X^+]<\infty\) and \(\mathbb E[X^-]<\infty\).

Theorem 1.3 (Jensen's inequality) Let \(I\subseteq\mathbb R\) be a convex set and \(f:I\to\mathbb R\) a convex function. Then for any random variable \(X\) with \(\mathbb P(x\in I)=1\), \(\mathbb E[|X|]<\infty\), \(\mathbb E[|f(X)|]<\infty\), $$f(\mathbb E[X])\le\mathbb E[f(X)]$$

Proof (Theorem 1.3) Let \(c=\mathbb E[X]\) (assume \(c\in\mathrm{int}(I)\); the boundary case is obvious). By convexity, the difference quotients \(\Delta_{+,h}(c)=\frac{f(c+h)-f(c)}{h}\), \(\Delta_{-,h}(c)=\frac{f(c)-f(c-h)}{h}\) are monotone and bounded as \(h\downarrow0\), with limits \(D_+(c)\ge D_-(c)\). Pick any \(m\in[D_-(c),D_+(c)]\) and define the supporting line \(L(x)\equiv f(c)+m(x-c)\). One shows \(L(x)\le f(x)\) always (split into \(x=c\), \(x>c\), \(x

Definition 1.4 / Proposition 1.1 The raw moment \(\mathbb E[X^k]\) is the \(k\)th raw moment of \(X\); the centered moment \(\mathbb E[(X-\mathbb E[X])^k]\) is the \(k\)th centered moment. Proposition 1.1: existence of a higher-order moment implies existence of a lower-order moment, i.e. for \(\forall1\le j\le k\), $$\mathbb E[|X|^k]<\infty\Rightarrow\mathbb E[|X|^j]<\infty$$

Proof (Proposition 1.1, truncation trick) Let \(1\le j\le k\) and \(f(x)=x^{k/j}\) (convex). Define \(Y=|X|^j\) and the truncation \(Y_n=\min\{|X|^j,n\}\). First show \(\mathbb E[|f(Y_n)|]<\infty\) and \(\mathbb E[||Y_n||]<\infty\). Since \(Y_n\le Y\), \(f\) increasing, \(f(Y_n)\ge0\): $$\mathbb E[|f(Y_n)|]\le\mathbb E[|f(Y)|]=\mathbb E[(|X|^j)^{k/j}]=\mathbb E[|X|^k]<\infty \tag{1.3}$$ Then, using \(f(x)>x\) (\(x>1\)) and \(f(x)\le x\) (\(0\le x\le1\)), split \(\mathbb E[|Y_n|]\) into two parts to show it is bounded \(<\infty\). As \(Y_n\to Y\) and expectation is continuous, \(\mathbb E[|X^j|]=\mathbb E[|Y|]<\infty\). \(\blacksquare\)

Theorem 1.4 Let random-vector sequences \(X_n(X_{n,1},\dots,X_{n,k})\) and \(X(X_1,\dots,X_k)\) both be in \(\mathbb R^k\). If \(X_{n,j}\xrightarrow{p}X_j\) for all \(1\le j\le k\), then \(X_n\xrightarrow{p}X\). (This generalizes the WLLN from one dimension to many.)

Proof (Theorem 1.4) By the definition of Euclidean distance and the union bound: $$\mathbb P(|X_n-X|>\varepsilon)=\mathbb P\Big(\sum_{i=1}^k(X_{n,i}-X_i)^2>\varepsilon^2\Big)\le\mathbb P\Big(\bigcup_{i=1}^k\{(X_{n,i}-X_i)^2>\tfrac{\varepsilon^2}{k}\}\Big)\le\sum_{i=1}^k\mathbb P\Big(|X_{n,i}-X_i|>\tfrac{\varepsilon}{\sqrt k}\Big)\to0$$ so \(X_n\xrightarrow{p}X\). \(\blacksquare\)

Theorem 1.5 (Continuous Mapping Theorem, CMT, in probability) Let \(\{X_n:n\ge1\}\) and \(X\) be random vectors on \(\mathbb R^k\), and \(g:\mathbb R^k\to\mathbb R^d\) be continuous at each point in a set \(C\subseteq\mathbb R^k\) with \(\mathbb P(X\in C)=1\). Then \(X_n\xrightarrow{p}X\Rightarrow g(X_n)\xrightarrow{p}g(X)\).

Proof (Theorem 1.5) For the bad set of points where \(X_n\) and \(X\) are close but \(g(X_n),g(X)\) stay far apart, \(B_\delta\equiv\{x:\exists y,|x-y|<\delta\text{ s.t. }|g(x)-g(y)|>\varepsilon\}\), $$\mathbb P(|g(X_n)-g(X)|>\varepsilon)=\mathbb P(\{\dots\}\cap\{X\notin B_\delta\})+\mathbb P(\{\dots\}\cap\{X\in B_\delta\}) \tag{1.4}$$ The first term \(\le\mathbb P(|X_n-X|\ge\delta)\to0\); the second \(\le\mathbb P(X\in B_\delta)\to0\) (\(\delta\) arbitrarily small). So \(g(X_n)\xrightarrow{p}g(X)\). \(\blacksquare\)

Remark 1.1 Using the CMT requires checking continuity at the point of interest. Counterexample: \(g(x)=\mathbf 1\{x>0\}\), take \(X_n=\frac1n,X=0\); though \(X_n\xrightarrow{p}X\), we have \(g(X_n)=1\not\xrightarrow{p}0=g(X)\) (\(g\) is discontinuous at \(0\)).

Definitions 1.5 / 1.6 An estimator \(\hat\mu(P)\) is a consistent estimator of \(\mu(P)\) if \(\hat\mu(P)\xrightarrow{p}\mu(P)\); an unbiased estimator if \(\mathbb E[\hat\mu(P)]=\mu(P)\).

Remark 1.2 Unbiasedness and consistency are closely related but not equivalent: unbiasedness is not asymptotic and does not imply consistency; consistency does not imply unbiasedness.

Definition 1.7 / Proposition 1.2 / Proposition 1.3 Definition 1.7 (convergence in \(q\)th moment): \(X_n\xrightarrow{L^q}X\) (\(q\ge1\)) if \(\mathbb E[|X_n-X|^q]\to0\). Proposition 1.2: convergence in moments ⇒ convergence in probability (\(X_n\xrightarrow{L^q}X\Rightarrow X_n\xrightarrow{p}X\)). Proposition 1.3: convergence in higher moments ⇒ in lower moments, i.e. for \(1\le j\le k\), \(\mathbb E[|X_n-X|^k]\to0\Rightarrow\mathbb E[|X_n-X|^j]\to0\).

Proof (Propositions 1.2 and 1.3) Prop 1.2: by Markov, \(\mathbb P(|X_n-X|>\varepsilon)\le\frac{\mathbb E[|X_n-X|^q]}{\varepsilon^q}\to0\). \(\blacksquare\) Prop 1.3: suppose \(\mathbb E[|X_n-X|^k]\to0\), so for large \(n\), \(\mathbb E[|X_n-X|^k]<\infty\). Take the concave \(f(x)=x^{j/k}\); Proposition 1.1 gives \(\mathbb E[|X_n-X|^j]<\infty\). By Jensen (reversed for concave): \(0\le\mathbb E[|X_n-X|^j]=\mathbb E[f(|X_n-X|^k)]\le f(\mathbb E[|X_n-X|^k])=(\mathbb E[|X_n-X|^k])^{j/k}\to0\). \(\blacksquare\)

Definition 1.8 (Convergence in distribution) \(X_n\xrightarrow{d}X\) if \(\mathbb P(X_n\le x)\to\mathbb P(X\le x)\) for all \(x\) at which \(\mathbb P(X\le x)\) is continuous. (Also called convergence in law, or weak convergence.)

Theorem 1.6 (Portmanteau lemma) The following characterizations of convergence in distribution are equivalent: (1) \(X_n\xrightarrow{d}X\); (2) \(\mathbb E[f(X_n)]\to\mathbb E[f(X)]\) for all bounded continuous \(f\); (3) for all bounded Lipschitz \(f\); (4) for all bounded uniformly continuous \(f\); (5) \(\liminf\mathbb P(X_n\in G)\ge\mathbb P(X\in G)\) for open \(G\); (6) \(\limsup\mathbb P(X_n\in F)\le\mathbb P(X\in F)\) for closed \(F\); (7) \(\mathbb P(X_n\in B)\to\mathbb P(X\in B)\) for all Borel sets \(B\) with \(\mathbb P(X\in\partial B)=0\).

Remark 1.3 Convergence in probability is about realizations (two random variables' realizations can be far apart), while convergence in distribution is only about the distribution, with nothing to do with realizations.

Theorem 1.7 / Lemma 1.4 Theorem 1.7 (CMT, in distribution): if \(g\) is continuous on \(C\) and \(\mathbb P(X\in C)=1\), then \(X_n\xrightarrow{d}X\Rightarrow g(X_n)\xrightarrow{d}g(X)\). Lemma 1.4 (Slutsky's lemma): if \(X_n\xrightarrow{d}X\) and \(Y_n\xrightarrow{d}c\) (a constant), then \(X_n'Y_n\xrightarrow{d}X'c\) and \(X_n+Y_n\xrightarrow{d}X+c\).

Definitions 1.9 / 1.10 Almost sure convergence (a.s.): \(X_n\xrightarrow{a.s.}X\) if \(\mathbb P(\lim_{n\to\infty}X_n=X)=1\), equivalently \(\mathbb P(\{\omega\in\Omega:\lim X_n(\omega)=X(\omega)\})=1\). Point-wise (surely) convergence: \(\lim X_n(\omega)=X(\omega)\) for all \(\omega\in\Omega\), equivalently \(\{\omega:\lim X_n(\omega)=X(\omega)\}=\Omega\).

Proposition 1.6 (Order of convergence) $$X_n\xrightarrow{p.w.}X\;\Rightarrow\;X_n\xrightarrow{a.s.}X\;\Rightarrow\;X_n\xrightarrow{p}X\;\Rightarrow\;X_n\xrightarrow{d}X$$ and $$X_n\xrightarrow{L^q}X\;\Rightarrow\;X_n\xrightarrow{p}X\;\Rightarrow\;X_n\xrightarrow{d}X$$

Proof (Proposition 1.6, the a.s. ⇒ p step) Point-wise ⇒ a.s. is trivial; a.s. ⇒ p is the key step. Suppose \(X_n\xrightarrow{a.s.}X\), i.e. \(\mathbb P(\omega\in\mathcal A)=0\) with \(\mathcal A\equiv\{\omega:\lim X_n(\omega)\ne X(\omega)\}\). For fixed \(\varepsilon>0\) define \(M_n=\bigcup_{m\ge n}\{|X_m-X|>\varepsilon\}\), so \(M_n\supseteq M_{n+1}\supseteq\dots\) and \(\mathbb P(\omega\in M_n)\) decreases; let \(M_\infty=\bigcap_{j=1}^\infty M_j\). Since \(\Omega\backslash\mathcal A\subseteq M_\infty^c\), we get \(M_\infty\subseteq\mathcal A\), so \(\mathbb P(\omega\in M_\infty)=0\). Hence $$\lim_{n\to\infty}\mathbb P(|X_n-X|>\varepsilon)\le\lim_{n\to\infty}\mathbb P(\omega\in M_n)=\mathbb P(\omega\in M_\infty)=0 \tag{1.6}$$ i.e. almost sure convergence yields convergence in probability. \(\blacksquare\)

Theorem 1.8 / Lemma 1.5 / Theorem 1.9 Theorem 1.8 (Univariate CLT): \(X_1,\dots,X_n\) i.i.d. on \(\mathbb R\) with \(\sigma^2(P)<\infty\). Then $$\sqrt n(\bar X_n-\mu(P))\xrightarrow{d}N(0,\sigma^2(P))$$ Lemma 1.5 (Cramér-Wold): \(X_n\xrightarrow{d}X\) iff \(t'X_n\xrightarrow{d}t'X\) for all \(t\in\mathbb R^k\). Theorem 1.9 (Multivariate CLT): \(X_1,\dots,X_n\) i.i.d. on \(\mathbb R^k\) with \(\Sigma(P)<\infty\). Then $$\sqrt n(\bar X_n-\mu(P))\xrightarrow{d}N(0,\Sigma(P))$$

Proof (Theorem 1.9, reducing to the univariate case via Cramér-Wold) It suffices to show the projection \(t'\sqrt n(\bar X_n-\mu)\xrightarrow{d}N(0,\Sigma(P))\). We have \(\mathbb E[t'X_i]=t'\mu(P)\), \(\mathrm{Var}[t'X_i]=t'\Sigma(P)t<\infty\), and \(t'X_i\) is still i.i.d.; apply the univariate CLT: $$\sqrt n\Big(\tfrac1n\sum t'X_i-t'\mu(P)\Big)\xrightarrow{d}N(0,\mathrm{Var}(t'X_i))=N(0,t'\Sigma(P)t)$$ i.e. \(t'\sqrt n(\bar X_n-\mu(P))\xrightarrow{d}t'N(0,\Sigma(P))\). By Cramér-Wold, \(\sqrt n(\bar X_n-\mu(P))\xrightarrow{d}N(0,\Sigma(P))\). \(\blacksquare\)

Definition 1.11 (Consistency in level) A test \(\phi_n\in[0,1]\) is consistent in level \(\alpha\) if $$\limsup_{n\to\infty}\mathbb E_P[\phi_n]\le\alpha$$ for \(P\) satisfying \(H_0\), where \(\alpha\in(0,1)\) is the significance level. Often \(\phi_n=\mathbf 1\{T_n>c_n\}\) with test statistic \(T_n\) and critical value \(c_n\).

Definition 1.12 (\(p\)-value) In a test, the \(p\)-value is the smallest \(\alpha\) at which \(H_0\) can be rejected consistently at level \(\alpha\): $$\hat p_n\equiv\inf\{\alpha\in(0,1):T_n>c_n\}$$

Definition 1.13 (Confidence region) A confidence region \(C_n=C_n(X_1,\dots,X_n)\) of level \(1-\alpha\) satisfies that the probability the true value lies in the set is no smaller than \(1-\alpha\): $$\mathbb P(\mu(P)\in C_n)\ge1-\alpha$$

Remark 1.4 (a useful fact) \(x\sim N(\mu_x,\Sigma(P))\Rightarrow(x-\mu_x)'\Sigma^{-1}(P)(x-\mu_x)\sim\chi^2_k\) (\(\Sigma(P)\) invertible, \(x\) a \(k\times1\) vector). Proof: by the multivariate-normal definition \(x-\mu_x=Az\), \(z\sim N(0,\mathbf I)\), \(\Sigma(P)=AA'\); then \((x-\mu_x)'\Sigma^{-1}(P)(x-\mu_x)=z'z\sim\chi^2_k\).

Theorem 1.10 (Delta method) Let \(\{X_n\}\), \(X\) be random vectors on \(\mathbb R^k\), \(c\in\mathbb R^k\) a constant vector, and \(\tau_n\to\infty\) a non-random sequence with \(\tau_n(X_n-c)\xrightarrow{d}X\); let \(g:\mathbb R^k\to\mathbb R^d\) be continuously differentiable at \(c\) with Jacobian \(D_g(c)\equiv\frac{\partial g(x)}{\partial x'}\big|_{x=c}\) (\(d\times k\)). Then $$\tau_n(g(X_n)-g(c))\xrightarrow{d}D_g(c)X$$ In particular, when \(X\sim N(0,\Sigma)\), \(\tau_n(g(X_n)-g(c))\xrightarrow{d}N(0,D_g(c)\Sigma D_g(c)')\).

Proof (Theorem 1.10, first-order Taylor expansion) First-order Taylor expansion of \(g(c)\) around \(c\): $$g(X_n)=g(c)+D_g(c)(X_n-c)+R(X_n-c) \tag{1.10}$$ with remainder \(R(0)=0\) and \(\frac{R(X_n-c)}{|X_n-c|}\to0\) (1.11). Multiply by \(\tau_n\) and rearrange: \(\tau_n(g(X_n)-g(c))=D_g(c)\tau_n(X_n-c)+\tau_n R(X_n-c)\), where \(\tau_n R(X_n-c)=\tau_n|X_n-c|\cdot\frac{R(X_n-c)}{|X_n-c|}\xrightarrow{d}|X|\times0=0\) (Slutsky). So \(\tau_n(g(X_n)-g(c))\xrightarrow{d}D_g(c)X\). \(\blacksquare\)

Definition 1.14 / Theorem 1.11 / Proposition 1.7 Definition 1.14 (correlation): when \(\mathbb E[X_i^2]<\infty,\mathbb E[Y_i^2]<\infty\), \(\mathrm{Var}(X_i)>0,\mathrm{Var}(Y_i)>0\), \(\rho_{X_i,Y_i}(P)\equiv\frac{\mathrm{Cov}(X_i,Y_i)}{\sqrt{\mathrm{Var}(X_i)}\sqrt{\mathrm{Var}(Y_i)}}\) (measures the linear relationship). Theorem 1.11 (Cauchy-Schwarz): for random variables \(u,v\) (\(\mathbb E[u^2],\mathbb E[v^2]<\infty\)), \(\mathbb E[uv]^2\le\mathbb E[u^2]\mathbb E[v^2]\) (1.12), strict unless \(\exists\alpha,\mathbb P(u=\alpha v)=1\). Proposition 1.7: \(|\rho_{X_i,Y_i}(P)|\le1\), with equality iff \(\exists a,b,\mathbb P(Y_i=a+bX_i)=1\).

Proof (Theorem 1.11, Cauchy-Schwarz) The trivial case \(\mathbb E[u^2]\) or \(\mathbb E[v^2]=0\) is obvious. Otherwise consider \(\mathbb E[(u-\alpha v)^2]=\mathbb E[u^2]-2\alpha\mathbb E[uv]+\alpha^2\mathbb E[v^2]\ge0\), a quadratic in \(\alpha\) minimized at \(\alpha=\frac{\mathbb E[uv]}{\mathbb E[v^2]}\); substituting gives \(\mathbb E[u^2]-\frac{\mathbb E[uv]^2}{\mathbb E[v^2]}\ge0\), i.e. \(\mathbb E[u^2]\mathbb E[v^2]\ge\mathbb E[uv]^2\). Strict unless \(\mathbb P(u-\alpha v=0)=1\). \(\blacksquare\)

Theorem 1.12 / Proposition 1.8 Theorem 1.12 (Berry-Esseen CLT): \(X_1,\dots,X_n\) i.i.d., \(\sigma^2(P)<\infty\); then uniformly \(\sup_{x}|\mathbb P(\frac{\sqrt n(\bar X_n-\mu(P))}{\sigma(P)}\le x)-\Phi(x)|\le\frac c{\sqrt n}\frac{\mathbb E[|X_i-\mu(P)|^3]}{\sigma^3(P)}\). Proposition 1.8 (limiting distribution of the median): if \(F\) is continuously differentiable at \(\theta\) with p.d.f. \(f\), then $$\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}N\Big(0,\frac1{4f^2(\theta)}\Big)$$

Proof idea (Proposition 1.8) The delta method does not apply (\(\theta\) is not a smooth function). Assume \(n\) even, \(\hat\theta_n\) the \(\frac n2\)th highest \(X_i\). Define \(Z_i\equiv\mathbf 1\{X_i>\theta+\frac x{\sqrt n}\}\), \(\mu_n=\mathbb E[Z_i]=1-F(\theta+\frac x{\sqrt n})\to\frac12\), \(\sigma_n^2=\mu_n(1-\mu_n)\to\frac14\). By Berry-Esseen, \(\mathbb P(\frac{\sqrt n(\bar Z_n-\mu_n)}{\sigma_n}\le z_n)\to\Phi(z_n)\), where \(z_n=\frac{\sqrt n(\frac12-\mu_n)}{\sigma_n}\to\frac{xf(\theta)}{\frac12}=2xf(\theta)\) (1.14). So \(\mathbb P(\sqrt n(\hat\theta_n-\theta)\le x)\to\Phi(2xf(\theta))\), i.e. \(\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}N(0,\frac1{4f^2(\theta)})\). \(\blacksquare\)

Definition 1.15 / Proposition 1.9 / Theorem 1.13 Definition 1.15 (tightness): a sequence \(\{X_n:n\ge1\}\) is tight if for any \(\varepsilon>0\) there exists a finite constant \(B>0\) with \(\inf_n\mathbb P(|X_n|\le B)\ge1-\varepsilon\). Proposition 1.9: convergence in distribution ⇒ tight. Theorem 1.13 (Prokhorov's theorem): if \(\{X_n:n\ge1\}\) is tight, there exist a subsequence \(\{X_{n_j}\}\) and a random vector \(X\) with \(X_{n_j}\xrightarrow{d}X\).

Definition 1.16 / Proposition 1.10 Definition 1.16 (\(\tau_n\)-consistency): if \(\tau_n(\hat\theta_n-\theta(P))\) is tight (with \(\tau_n\to\infty\)), then \(\hat\theta_n\) is a \(\tau_n\)-consistent estimator of \(\theta(P)\). Proposition 1.10: \(\tau_n\)-consistent ⇒ consistent.

Proof (Proposition 1.10) If the estimation error blown up by \(\tau_n\) (which goes to \(\infty\)) is still bounded (tight), then the un-blown-up error must converge to \(0\). \(\blacksquare\)

Remark 1.7 \(o_P(1)\Rightarrow O_P(1)\) (convergence in probability to 0 ⇒ tight), but not conversely (\(O_P(1)\not\Rightarrow o_P(1)\)).

Proposition 1.11 (Rules of calculus with stochastic order) 1. \(o_P(1)+o_P(1)=o_P(1)\); 2. \(o_P(1)+O_P(1)=O_P(1)\); 3. \(o_P(1)O_P(1)=o_P(1)\); 4. \((1+o_P(1))^{-1}=O_P(1)\); 5. \(o_P(O_P(1))=o_P(1)\).

Proof highlights (Proposition 1.11) (1) \(X_n,Y_n\xrightarrow{p}0\), so by CMT \(X_n+Y_n\xrightarrow{p}0\). (2) \(X_n=o_P(1)\), \(Y_n=O_P(1)\) tight; use the union bound to show \(X_n+Y_n\) tight. (3) By contradiction + Prokhorov + Slutsky. (4) \(f(x)=\frac1{1+x}\) is continuous at \(x\ne-1\), so by CMT \(f(X_n)\xrightarrow{p}f(0)=1\) (convergence in probability ⇒ tight), hence \(f(X_n)=O_P(1)\). (5) A restatement of rule 3. \(\blacksquare\)

Chapter arc From "notions of convergence" to "statistical inference." §1.2–1.6 build the four types of convergence (in probability, in moments, in distribution, almost sure) and their hierarchy (p.w.⇒a.s.⇒p⇒d, \(L^q\)⇒p), with the core tools being Markov's inequality, Jensen's inequality, the Continuous Mapping Theorem, Slutsky's lemma, and the Portmanteau lemma. The CLT of §1.7 delivers asymptotic normality after \(\sqrt n\)-standardization. §1.8 applies these tools to inference: testing (consistency in level, \(p\)-value, multidimensional \(\chi^2\) test, delta method) and confidence regions (CLT vs Markov constructions). The tightness and stochastic-order \(o_P/O_P\) of §1.9–1.10 are the language for the asymptotic expansions of estimators in later chapters.