2. Conditional Expectation

Chapter theme: conditional expectation. Define the conditional expectation \(\mathbb E[Y\mid\mathbf X]\) as the best predictor in the \(L^2\) sense — the function of \(\mathbf X\) (among all square-integrable ones) that minimizes the mean-squared distance to \(Y\). §2.1 Definition: when \(\mathbb E[Y^2]<\infty\), \(\mathbb E[Y\mid\mathbf X]=\arg\min_{m\in\mathbb M}\mathbb E[(Y-m(\mathbf X))^2]\) (2.1); Theorem 2.1 (orthogonality condition I) gives the equivalent characterization \(\mathbb E[(Y-m^*(\mathbf X))m(\mathbf X)]=0,\forall m\in\mathbb M\) (2.2) and proves uniqueness. When only \(\mathbb E[|Y|]<\infty\), Theorem 2.2 (orthogonality condition II): \(\mathbb E[(Y-m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=0\) for any Borel set (2.3). §2.2 Properties (Proposition 2.1, seven items): pulling out functions, linearity, pulling out \(f(\mathbf X)\), sign preservation, the law of iterated expectations (LIE) \(\mathbb E[Y]=\mathbb E[\mathbb E[Y\mid\mathbf X]]\), independence ⇒ \(\mathbb E[Y\mid\mathbf X]=\mathbb E[Y]\), and mean independence. The implication chain: independence ⇒ mean independence ⇒ uncorrelatedness, with neither arrow reversible (counterexamples \(Y=X^2\) and \(Y\mid X\sim N(0,\sigma^2X)\)).

Definition 2.1 (Conditional expectation) The conditional expectation of \(Y\) given \(\mathbf X\), \(\mathbb E[Y\mid\mathbf X]\), is any solution to: $$\inf_{m\in\mathbb M}\mathbb E[(Y-m(\mathbf X))^2] \tag{2.1}$$

Theorem 2.1 (Orthogonality condition I) For a random vector \((Y,\mathbf X)\), \(Y\in\mathbb R\), \(\mathbf X\in\mathbb R^k\), \(\mathbb E[Y^2]<\infty\), then \(m^*(\mathbf X)\in\mathbb M\) solves (2.1) if and only if $$\mathbb E[(Y-m^*(\mathbf X))m(\mathbf X)]=0,\quad\forall m(\mathbf X)\in\mathbb M \tag{2.2}$$ and \(m^*\) is unique in the sense that if \(\tilde m(\mathbf X)\) also solves (2.1), then \(\mathbb P(\tilde m(\mathbf X)=m^*(\mathbf X))=1\).

Proof (Theorem 2.1, orthogonality ⟺ minimization) (⇐) Orthogonality ⇒ minimization. Take any \(m\in\mathbb M\) and add/subtract \(m^*(\mathbf X)\): $$\mathbb E[(Y-m(\mathbf X))^2]=\mathbb E[(Y-m^*(\mathbf X))^2]+2\mathbb E[(Y-m^*(\mathbf X))(m^*(\mathbf X)-m(\mathbf X))]+\mathbb E[(m^*(\mathbf X)-m(\mathbf X))^2]$$ By Cauchy-Schwarz, \(\mathbb E[(m^*(\mathbf X)-m(\mathbf X))^2]<\infty\), so \(m_0(\mathbf X)\equiv m^*(\mathbf X)-m(\mathbf X)\in\mathbb M\), and the cross term $=0$ by the orthogonality condition. Hence $$\mathbb E[(Y-m(\mathbf X))^2]=\mathbb E[(Y-m^*(\mathbf X))^2]+\mathbb E[(m^*(\mathbf X)-m(\mathbf X))^2]\ge\mathbb E[(Y-m^*(\mathbf X))^2]$$ so \(m^*\) solves (2.1); the inequality is strict unless \(\mathbb P(m(\mathbf X)=m^*(\mathbf X))=1\), giving uniqueness. (⇒) Minimization ⇒ orthogonality. Suppose \(m^*\) solves (2.1); then for all \(\alpha\in\mathbb R\), $$\mathbb E[(Y-m^*(\mathbf X))^2]\le\mathbb E[(Y-m^*(\mathbf X)-\alpha m(\mathbf X))^2]=\mathbb E[(Y-m^*(\mathbf X))^2]+\alpha^2\mathbb E[(m(\mathbf X))^2]-2\alpha\mathbb E[(Y-m^*(\mathbf X))m(\mathbf X)]$$ i.e. \(\alpha^2\mathbb E[(m(\mathbf X))^2]-2\alpha\mathbb E[(Y-m^*(\mathbf X))m(\mathbf X)]\ge0\) for all \(\alpha\). For this quadratic in \(\alpha\) to be nonnegative, the linear coefficient must vanish: \(\mathbb E[(Y-m^*(\mathbf X))m(\mathbf X)]=0\) (orthogonality condition I). \(\blacksquare\)

Theorem 2.2 (Orthogonality condition II) For a random vector \((Y,\mathbf X)\), \(Y\in\mathbb R\), \(\mathbf X\in\mathbb R^k\), \(\mathbb E[|Y|]<\infty\), the conditional expectation of \(Y\) given \(\mathbf X\), \(\mathbb E[Y\mid\mathbf X]\), is any \(m^*(\mathbf X)\) with \(\mathbb E[|m^*(\mathbf X)|]<\infty\) such that for any Borel set \(\mathbf B\) in \(\mathbb R^k\), $$\mathbb E[(Y-m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=0 \tag{2.3}$$

Proposition 2.1 (Properties of conditional expectation) 1. If \(Y=f(\mathbf X)\), then \(\mathbb E[Y\mid\mathbf X]=f(\mathbf X)\). 2. (Linearity) \(\mathbb E[Y+Z\mid\mathbf X]=\mathbb E[Y\mid\mathbf X]+\mathbb E[Z\mid\mathbf X]\). 3. (Pulling out a known function) \(\mathbb E[f(\mathbf X)Y\mid\mathbf X]=f(\mathbf X)\mathbb E[Y\mid\mathbf X]\). 4. (Sign preservation) If \(\mathbb P(Y\ge0)=1\), then \(\mathbb P(\mathbb E[Y\mid\mathbf X]\ge0)=1\). 5. (Law of iterated expectations, LIE) Taking \(\mathbf B=\mathbb R^k\), (2.3) implies \(\mathbb E[Y-\mathbb E[Y\mid\mathbf X]]=0\), i.e. \(\mathbb E[Y]=\mathbb E[\mathbb E[Y\mid\mathbf X]]\). More generally \(\mathbb E[\mathbb E[Y\mid\mathbf X_1,\mathbf X_2]\mid\mathbf X_1]=\mathbb E[Y\mid\mathbf X_1]\). 6. (Independence) If \(\mathbf X\perp Y\), then \(\mathbb E[Y\mid\mathbf X]=\mathbb E[Y]\). 7. (Mean independence) If \(\mathbb E[Y\mid\mathbf X]=c\) (a constant), then by LIE \(\mathbb E[\mathbb E[Y\mid\mathbf X]]=\mathbb E[c]=c\), i.e. \(\mathbb E[Y\mid\mathbf X]=\mathbb E[Y]\).

Proof (Properties 1, 2) Property 1: for this type of problem, first "guess" a good \(m^*(\mathbf X)\) and plug it into the orthogonality condition. Try \(m^*(\mathbf X)=f(\mathbf X)\): $$\mathbb E[(Y-m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=\mathbb E[(f(\mathbf X)-f(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=0$$ for all \(\mathbf B\). By uniqueness of \(\mathbb E[Y\mid\mathbf X]\), \(\mathbb E[Y\mid\mathbf X]=m^*(\mathbf X)=f(\mathbf X)\). Property 2: let \(\mathbb E[Y\mid\mathbf X]=m^*(\mathbf X)\) satisfy (2.3) and \(\mathbb E[Z\mid\mathbf X]=\tilde m^*(\mathbf X)\) satisfy (2.3). By linearity of expectation, \(\mathbb E[(Y-m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]+\mathbb E[(Z-\tilde m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=\mathbb E[(Y+Z-m^*(\mathbf X)-\tilde m^*(\mathbf X))\mathbf 1\{\mathbf X\in\mathbf B\}]=0\) for all \(\mathbf B\), so \(\mathbb E[Y+Z\mid\mathbf X]=m^*(\mathbf X)+\tilde m^*(\mathbf X)=\mathbb E[Y\mid\mathbf X]+\mathbb E[Z\mid\mathbf X]\). \(\blacksquare\)

Proof (implication chain and counterexamples) Independence ⇒ mean independence: verify that \(m^*(\mathbf X)=\mathbb E[Y\mid\mathbf X]=\mathbb E[Y]\) satisfies (2.3): $$\mathbb E[(Y-\mathbb E[Y])\mathbf 1\{\mathbf X\in\mathbf B\}]\overset{\text{indep}}=\mathbb E[(Y-\mathbb E[Y])]\,\mathbb E[\mathbf 1\{\mathbf X\in\mathbf B\}]=\underbrace{(\mathbb E[Y]-\mathbb E[Y])}_{=0}\mathbb E[\mathbf 1\{\mathbf X\in\mathbf B\}]=0$$ for all \(\mathbf B\), so \(\mathbb E[Y\mid\mathbf X]=\mathbb E[Y]\). Mean independence ⇒ uncorrelatedness: suppose \(\mathbb E[Y\mid X]=\mathbb E[Y]\); then $$\mathrm{Cov}(X,Y)=\mathbb E[XY]-\mathbb E[X]\mathbb E[Y]\overset{\text{LIE}}=\mathbb E[\mathbb E[XY\mid X]]-\mathbb E[X]\mathbb E[Y]\overset{\text{Prop 3}}=\mathbb E[X\mathbb E[Y\mid X]]-\mathbb E[X]\mathbb E[Y]=\mathbb E[X]\mathbb E[Y]-\mathbb E[X]\mathbb E[Y]=0$$ Uncorrelatedness ⇏ mean independence (counterexample): \(X\sim N(0,1)\), \(Y\overset d=X^2\). Then \(\mathrm{Cov}(X,Y)=\mathbb E[X^3]-\mathbb E[X]\mathbb E[X^2]=0-0=0\) (uncorrelated), but \(\mathbb E[Y\mid X]=\mathbb E[X^2\mid X]=X^2\ne\mathbb E[Y]=1\) (not mean independent). Mean independence ⇏ independence (counterexample): \(Y\mid X\sim N(0,\sigma^2X)\), so \(\mathbb E[Y\mid X]=0\) is constant (mean independent), but since the variance of \(Y\mid X\) depends on \(X\), \(Y\) is not independent of \(X\). \(\blacksquare\)

Summary The core of conditional expectation is the geometric view of "projection / best prediction": \(\mathbb E[Y\mid\mathbf X]\) is the \(L^2\) projection of \(Y\) onto the space of \(\mathbf X\)-measurable functions, with the residual orthogonal to all \(m(\mathbf X)\) (orthogonality conditions I/II). The next chapter's three interpretations of linear regression are exactly the special case of shrinking this projection from "the space of arbitrary functions \(\mathbb M\)" to "the space of linear functions of \(\mathbf X\)." The chain independence ⇒ mean independence ⇒ uncorrelatedness and its irreversibility are the basis for later distinguishing strong vs weak "exogeneity."