21. Multivariate Normal Distribution

Chapter theme: the multivariate normal distribution. The multivariate (joint) normal distribution generalizes the univariate normal to higher dimensions; its key feature is that any linear combination of all components is a univariate normal distribution — crucial to multivariate statistical inference. §21.1 Definition: Definition 21.1 standard normal random vector (components i.i.d. standard normal); Definition 1 (Definition 21.2) \(\mathbf X=\mathbf A\mathbf Z+\mu\) (there exist a standard normal vector \(\mathbf Z\) and a matrix \(\mathbf A\)); Definition 2 (Definition 21.3) for any \(\mathbf b\), \(\mathbf b'\mathbf X\) is univariate normal. §21.2 Equivalence of the two definitions: Definition 21.4 convolution \((f*g)(x)=\int f(t)g(x-t)\,dt\); Lemma 21.1 any linear combination of independent normals is normal (proved by induction via convolution); hence the two definitions are equivalent. §21.3 Density of the multivariate normal: \(f_{\mathbf X}(\mathbf x)=\frac{1}{\sqrt{(2\pi)^k|\Omega|}}e^{-\frac12(\mathbf x-\mu)'\Omega^{-1}(\mathbf x-\mu)}\) (since \((\mathbf x-\mu)'\Omega^{-1}(\mathbf x-\mu)\sim\chi_k^2\)).

Definition 21.1 (Standard normal random vector) A real random vector \(\mathbf Z=(Z_1,Z_2,\dots,Z_l)'\) is called a standard normal random vector if all of its components \(Z_n\), \(n=1,2,\dots,l\), are i.i.d. with univariate standard normal distribution.

Definition 21.2 (Multivariate normal distribution-1) A real random vector \(\mathbf X=(X_1,X_2,\dots,X_k)'\) is called a normal random vector and its components are multivariate normally distributed if and only if there exist a standard normal random vector \(\mathbf Z_{l\times1}\) and a matrix \(\mathbf A_{k\times l}\) such that $$\mathbf X=\mathbf A\mathbf Z+\mu$$ where \(\mu\) is the mean vector of \(\mathbf X\).

Definition 21.3 (Multivariate normal distribution-2) A real random vector \(\mathbf X=(X_1,X_2,\dots,X_k)'\) is called a normal random vector and its components are multivariate normally distributed if and only if any linear combination of its components is (univariate) normally distributed, i.e. for \(\forall\mathbf b=(b_1,\dots,b_k)'\in\mathbb R^k\), $$Y=\mathbf b'\mathbf X=b_1X_1+b_2X_2+\cdots+b_kX_k\sim\mathcal N(\mu_Y,\sigma_Y^2)$$

Definition 21.4 (Convolution) A convolution is defined as an operator $*$ for two functions \(f\) and \(g\) on \(\mathbb R\) such that $$(f*g)(x)\equiv\int_{-\infty}^{\infty}f(t)g(x-t)\,dt$$

Lemma 21.1 Any linear combination of a set of independent normally distributed random variables \((X_1,X_2,\dots,X_k)\) is normally distributed.

Proof The proof is by induction. We first consider any simple sum of two independent normally distributed variables \(X_i\sim\mathcal N(\mu_i,\sigma_i^2)\) and \(X_j\sim\mathcal N(\mu_j,\sigma_j^2)\). Suppose \(X_i\) has density function \(f_i(x_i)\) and \(X_j\) has density function \(f_j(x_j)\). Then denote \(Y=X_i+X_j\), whose density is \(g(y)\). Since \(X_i\) and \(X_j\) are independent, we have that $$\begin{aligned}g(y)&=(f_i*f_j)(y)\\&=\int_{-\infty}^{\infty}f_i(t)f_j(y-t)\,dt\\&=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma_i}e^{-\frac12\left(\frac{t-\mu_i}{\sigma_i}\right)^2}\frac{1}{\sqrt{2\pi}\sigma_j}e^{-\frac12\left(\frac{y-t-\mu_j}{\sigma_j}\right)^2}\,dt\end{aligned}$$ Completing the square in the exponent separates the integrand into a Gaussian kernel in \(t\) (of the form \(e^{-\frac12\left(\frac{t-\text{(linear in }y)}{\sigma_i\sigma_j/\sqrt{\sigma_i^2+\sigma_j^2}}\right)^2}\), which integrates over \(t\) to its normalizing constant) times a factor \(R\) not related to \(t\), where $$R=e^{-\frac12\left(\frac{y-\mu_i-\mu_j}{\sqrt{\sigma_j^2+\sigma_i^2}}\right)^2}$$ So, $$\begin{aligned}g(y)&=\frac{1}{\sqrt{2\pi}}R\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma_i\sigma_j}e^{-\frac12\left(\frac{t-\cdots}{\sigma_i\sigma_j/\sqrt{\sigma_j^2+\sigma_i^2}}\right)^2}\,dt\\&=\frac{1}{\sqrt{2\pi}\sqrt{\sigma_j^2+\sigma_i^2}}R\underbrace{\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma_i\sigma_j/\sqrt{\sigma_j^2+\sigma_i^2}}e^{-\frac12\left(\frac{t-\cdots}{\sigma_i\sigma_j/\sqrt{\sigma_j^2+\sigma_i^2}}\right)^2}\,dt}_{=1}\\&=\frac{1}{\sqrt{2\pi}\sqrt{\sigma_j^2+\sigma_i^2}}R\\&=\frac{1}{\sqrt{2\pi}\sqrt{\sigma_j^2+\sigma_i^2}}e^{-\frac12\left(\frac{y-\mu_i-\mu_j}{\sqrt{\sigma_j^2+\sigma_i^2}}\right)^2}\end{aligned}$$ Clearly, \(g(y)\) is the density of normal distribution \(\mathcal N(\mu_i+\mu_j,\sigma_i^2+\sigma_j^2)\). Therefore, the simple sum of any two independent normally distributed random variables also follows a normal distribution.

Then, based on this result, we can prove that any arbitrary linear combination \(V=aX_i+bX_j\), \(\forall a,b\in\mathbb R\) follows univariate normal distribution. This is obvious since we can denote $$W_i=aX_i\sim\mathcal N(\tilde\mu_i,\tilde\sigma_i^2)\quad\text{and}\quad W_j=bX_j\sim\mathcal N(\tilde\mu_j,\tilde\sigma_j^2)$$ where \(\tilde\mu_i=a\mu_i\), \(\tilde\mu_j=b\mu_j\), \(\tilde\sigma_i^2=a^2\sigma_i^2\), \(\tilde\sigma_j^2=b^2\sigma_j^2\). Since \(W_i\) and \(W_j\) are also independent with each other, the proof above for simple sum also applies here, which means that any linear combination of two independent normally distributed random variables is univariate normally distributed.

Finally, we can do the linear combination of \(V\) and another \(cX_k\) (\(c\in\mathbb R\)) and simply repeat this procedure to prove that any linear combination of \(\mathbf X\)'s elements yields a univariate normally distributed random variable. \(\blacksquare\)

Remark 21.1 The intuition for the convolution is that the probability density of \(Y=y\) should be the sum of the possibilities of all possible combinations of values of \(X_i\) and \(X_j\).

Proof (Equivalence of two definitions) By Definition 21.2, each element of \(\mathbf X\) is a linear combination of elements in \(\mathbf Z\), so any linear combination of elements in \(\mathbf X\) is again a linear combination of elements in \(\mathbf Z\) and thus, by lemma 21.1, is univariate normally distributed, which gives us Definition 21.3.

Also note that the proof of lemma 21.1 heavily depends on the independence of \(X_i\) and \(X_j\) and on their normal distribution. So, for the property described in 21.3 to hold, we need each element in \(\mathbf X\) to be a linear combination of some independent normally distributed random variables, which can be transformed to a linear combination of i.i.d. standard normally distributed random variables, which gives us Definition 21.2. \(\blacksquare\)