9. Univariate Time Series

Chapter theme: univariate time series. §9.1 Lag-operator calculus: the martingale-difference-sequence (MDS) assumption \(E_{t-1}[\varepsilon_t]=0\); with the lag operator \(L\) write AR(\(m\)) as \((1-\rho(L))y_t=\varepsilon_t\), MA(\(n\)) as \(y_t=\theta(L)\varepsilon_t\), ARMA, and the VAR(1) stacking; define autocovariances \(\Gamma_k\), covariance stationarity; compute AR/MA autocovariances via Yule–Walker. §9.2 Characteristic polynomial: \(p(\lambda)=\lambda^m(1-\rho(\lambda^{-1}))\), whose roots equal the eigenvalues of the stacked matrix \(B\); an AR(\(m\)) is covariance stationary \(\iff\) all \(|\lambda_i|<1\); inverting lag polynomials rewrites AR as MA(\(\infty\)). §9.3 Wold decomposition: any covariance-stationary series \(y_t=\mu_t+\sum c_j u_{t-j}\) (\(c_0=1\), \(u_t\) the one-step forecast errors). §9.4 Forecasting and impulse responses: \(P(y_{t+k}\mid \mathcal F_t)\), impulse response \(=c_j\), two methods for AR(\(m\)). §9.5 Spectral theory: Fourier transforms, Euler's formula, the spectral density \(s_x(\omega)=\frac1{2\pi}\sum\gamma_j e^{-i\omega j}\), lag operator and spectrum \(s_y=|h(e^{-i\omega})|^2 s_x\), the Blaschke factor and root flipping, the fundamental representation of MA(\(n\)). §9.6 Unit roots: unit roots, integration of order \(r\) (\(I(r)\)), the difference operator \(\Delta=1-L\).

Definition 9.1 (Martingale difference sequence) A sequence \(\{\varepsilon_t\}\) is a martingale difference sequence (MDS) if \(E_{t-1}[\varepsilon_t]=0\).

Definition 9.2 (Martingale) A sequence \(\{x_t\}\) is a martingale if \(E_{t-1}[x_t]=x_{t-1}\).

Remark If \(\{\varepsilon_t\}\) is an MDS then \(x_t=\sum_{s\le t}\varepsilon_s\) is a martingale. The MDS is weaker than i.i.d. (it allows certain dependence beyond conditional heteroskedasticity) but stronger than "uncorrelated" (it requires the conditional mean to vanish).

Definition 9.3 (\(k\)-th autocovariance) For a (demeaned) series \(y_t\), the \(k\)-th autocovariance is $$\Gamma_k=E[y_t y_{t-k}']$$ In the scalar case write \(\gamma_k\), and the autocorrelation is \(\rho_k=\gamma_k/\gamma_0\).

Definition 9.4 (Covariance stationary) \(y_t\) is covariance stationary (weakly stationary) if \(E[y_t]\) and all \(\Gamma_k\) do not depend on \(t\).

Definition 9.5 (Vectorization and Kronecker product) \(\operatorname{vec}(\cdot)\) stacks the columns of a matrix into a vector. The Kronecker product satisfies $$\operatorname{vec}(AXB)=(B'\otimes A)\operatorname{vec}(X)$$

Proof (vectorization identity) Let \(b_j\) be the \(j\)-th column of \(B\). The \(j\)-th column of \(AXB\) is \(AXb_j=A\big(\sum_k X_{\cdot k}b_{kj}\big)=\sum_k b_{kj}\,(A X_{\cdot k})\), where \(X_{\cdot k}\) is the \(k\)-th column of \(X\). Stacking the columns and collecting coefficients yields \((B'\otimes A)\operatorname{vec}(X)\). \(\blacksquare\)

Definition 9.6 (Characteristic polynomial) For the AR(\(m\)) of (9.1), the characteristic polynomial is $$p(\lambda)=\lambda^m\big(1-\rho(\lambda^{-1})\big)=\lambda^m-\rho_1\lambda^{m-1}-\dots-\rho_m$$

Lemma 9.1 The roots of \(p(\lambda)\) are exactly the eigenvalues of the stacked matrix \(B\).

Theorem 9.1 (Fundamental theorem of algebra) Every degree-\(m\) polynomial has exactly \(m\) roots over the complex field (counting multiplicity).

Corollary 9.1 (Factoring the lag polynomial) If the roots of \(p(\lambda)\) are \(\lambda_1,\dots,\lambda_m\), then $$1-\rho(L)=(1-\lambda_1 L)(1-\lambda_2 L)\cdots(1-\lambda_m L)$$

Proposition 9.1 (Stationarity criterion) An AR(\(m\)) is covariance stationary \(\iff\) all characteristic roots satisfy \(|\lambda_i|<1\).

Remarks 9.5–9.6 Convergence of the inversion is just a restatement of the stationarity condition \(|\lambda_i|<1\): roots inside the unit circle \(\iff\) exponentially decaying infinite-order MA weights \(\iff\) a covariance-stationary series.

Theorem 9.2 (Wold decomposition) Any (zero-mean) covariance-stationary series \(y_t\) admits a unique decomposition $$y_t=\mu_t+\sum_{j=0}^\infty c_j u_{t-j},\qquad c_0=1$$ where \(\mu_t\) is the deterministic component linearly predictable from the infinite past of \(y\), \(u_t=y_t-P(y_t\mid y_{t-1},y_{t-2},\dots)\) are the one-step forecast errors (mutually uncorrelated white noise), and \(\sum c_j^2<\infty\).

Proposition 9.2 (Wold for AR(\(m\))) For a covariance-stationary AR(\(m\)), the Wold decomposition is exactly the MA(\(\infty\)) representation obtained by expanding \((1-\rho(L))^{-1}\), with \(u_t=\varepsilon_t\).

Proof (Taylor expansion) Group \(e^{i\omega}=\sum_{k\ge0}\frac{(i\omega)^k}{k!}\) by parity of \(k\): the even terms with \(i^{2n}=(-1)^n\) give \(\sum(-1)^n\omega^{2n}/(2n)!=\cos\omega\); the odd terms with \(i^{2n+1}=i(-1)^n\) give \(i\sum(-1)^n\omega^{2n+1}/(2n+1)!=i\sin\omega\). Summing yields the result. \(\blacksquare\)

Fourier transform and its inverse Given an absolutely summable (non-stochastic) sequence \(\{x_j\}\) (\(\sum_j|x_j|<\infty\)), define $$\tilde x(\omega)\equiv\frac1{2\pi}\sum_{j=-\infty}^\infty x_j e^{-i\omega j}$$ The inverse transform is $$x_j=\int_{-\pi}^{\pi}\tilde x(\omega)\,e^{i\omega j}\,d\omega$$

Proof (inverse transform) Key lemma: for all \(k\in\mathbb Z\), \(k\ne0\), $$\int_{-\pi}^{\pi}e^{i\omega k}\,d\omega=\Big[\tfrac1k\sin(k\omega)\Big]_{-\pi}^{\pi}-i\Big[\tfrac1k\cos(k\omega)\Big]_{-\pi}^{\pi}=\tfrac2k\sin(k\pi)+0=0$$ (for \(k=0\) the integral equals \(2\pi\).) Hence $$\int_{-\pi}^{\pi}\tilde x(\omega)e^{i\omega j}\,d\omega=\int_{-\pi}^{\pi}\frac1{2\pi}\Big(\sum_m x_m e^{-i\omega m}\Big)e^{i\omega j}\,d\omega=\frac1{2\pi}\int_{-\pi}^{\pi}x_j\,d\omega+\underbrace{\frac1{2\pi}\int_{-\pi}^{\pi}\sum_{m\ne j}x_m e^{i\omega(j-m)}\,d\omega}_{=0}=x_j$$ \(\blacksquare\)

Remarks 9.12–9.13 Complex values are used to conveniently adjust both amplitude and initial phase of each cosine line at once, while the imaginary part cancels out under integration. \(A^\omega=|\tilde x(\omega)|\) measures the height (weight) of each frequency's (initial) cosine line, so \(|\tilde x(\omega)|\) is the weight of the information carried by each frequency.

Fourier transform of a lag operator Let \(y_t=h(L)x_t=\sum_{j=-\infty}^\infty h_j x_{t-j}\), \(h(L)=\sum_j h_j L^j\). Then $$\tilde y(\omega)=h\big(e^{-i\omega}\big)\tilde x(\omega)=2\pi\tilde h(\omega)\tilde x(\omega) \tag{9.8}$$ where \(\tilde h(\omega)=\frac1{2\pi}\sum_j h_j e^{-i\omega j}\). Filtering is convolution in the time domain and pointwise multiplication in the frequency domain.

Proof (Fourier transform of a lag operator) $$\begin{aligned}h(e^{-i\omega})\tilde x(\omega)&=\Big(\sum_j h_j e^{-i\omega j}\Big)\Big(\frac1{2\pi}\sum_t x_t e^{-i\omega t}\Big)\\&=\frac1{2\pi}\sum_t\sum_j h_j x_{t-j}\,e^{-i\omega t}\quad(\text{change of index})\\&=\frac1{2\pi}\sum_t\Big(\sum_j h_j x_{t-j}\Big)e^{-i\omega t}=\frac1{2\pi}\sum_t y_t e^{-i\omega t}=\tilde y(\omega)\end{aligned}$$ and note \(h(e^{-i\omega})=\sum_j h_j e^{-i\omega j}=2\pi\tilde h(\omega)\). \(\blacksquare\)

Definition 9.7 (Population spectrum) Let \(x_t\) be a zero-mean covariance-stationary series with \(\gamma_j=E[x_t x_{t-j}]=\gamma_{-j}\). The population spectrum is the Fourier transform of the autocovariances: $$s_x(\omega)\equiv\tilde\gamma(\omega)=\frac1{2\pi}\sum_{j=-\infty}^\infty\gamma_j e^{-i\omega j}$$

Proof (spectrum as a limiting expected modulus) $$\begin{aligned}E\big[\tilde x_T(\omega)\overline{\tilde x_T(\omega)}\big]&=\frac1{2T}\frac1{2\pi}E\Big[\sum_{k=-T}^T\sum_{i=-T}^T x_i x_k e^{-i\omega(i-k)}\Big]\\&=\frac1{2T}\frac1{2\pi}\sum_{j=-2T}^{2T}\gamma_j\,(2T+1-|j|)\,e^{-i\omega j}\\&=\frac1{2\pi}\Big(\frac1{2\pi}\sum_{j=-2T}^{2T}\gamma_j\,\frac{2T+1-|j|}{2T}\,e^{-i\omega j}\Big)\end{aligned}$$ As \(T\to\infty\) the weight \(\frac{2T+1-|j|}{2T}\to1\), so \(\lim_T E[\tilde x_T\overline{\tilde x_T}]=\frac1{2\pi}s_x(\omega)\), i.e. \(s_x(\omega)=2\pi\lim_T E[\tilde x_T(\omega)\overline{\tilde x_T(\omega)}]\). \(\blacksquare\)

Remarks 9.14–9.17 - \(s_x(\omega)\) is real-valued, since \(\gamma_j=\gamma_{-j}\). - \(s_x(\omega)\propto E[|\tilde x(\omega)|^2]\) is a good measure of the (expected) information weight at each frequency \(\omega\); the expectation is taken because \(|\tilde x(\omega)|^2\) is random (\(x_t\) is random). - \(\tilde x(\omega)\) decomposes the random sequence \(x_t\) into frequency components, and \(s_x(\omega)\) measures the expected weight of each component. - From \(\gamma_0=\int_{-\pi}^{\pi}s_x(\omega)\,d\omega\), the spectrum decomposes the variance across frequencies: \(s_x(\omega)\) is the contribution of the frequency-\(\omega\) component to total variance.

Definition 9.8 (White noise) A sequence \(\epsilon_t\) is white noise if it is a martingale difference sequence with constant variance \(\sigma^2\).

Remark 9.18 The spectrum measures the expected weight of each frequency component; white noise's spectrum is the constant \(\frac{\sigma^2}{2\pi}\), meaning white noise puts equal weight on every frequency — frequencies are equally mixed, just as equally mixing different colors of light gives white light, hence "white" noise.

Proposition (spectrum after filtering) Let \(y_t=h(L)x_t=\sum_{j}h_j x_{t-j}\), \(h(L)=\sum_j h_j L^j\). Then $$s_y(\omega)=h\big(e^{-i\omega}\big)h\big(e^{i\omega}\big)s_x(\omega)=\big|h(e^{-i\omega})\big|^2 s_x(\omega) \tag{9.9}$$

Proof (rearranging autocovariances) $$\begin{aligned}s_y(\omega)&=\frac1{2\pi}\sum_j\gamma^y_j e^{-i\omega j}=\frac1{2\pi}\sum_j\sum_t\sum_k h_t h_{t-k}\gamma^x_{j+k}e^{-i\omega j}\\&=\frac1{2\pi}\sum_t\sum_k h_t h_{t-k}\Big(\sum_j\gamma^x_{j}e^{-i\omega(j-k)}\Big)\\&=\frac1{2\pi}\Big(\sum_t h_t e^{-i\omega t}\Big)\Big(\sum_k h_{t-k}\cdots\Big)\Big(\sum_j\gamma^x_j e^{-i\omega j}\Big)\\&=h(e^{-i\omega})h(e^{i\omega})s_x(\omega)\end{aligned}$$ Rearranging the order of summation separates out the two filter factors \(\sum h_j e^{-i\omega j}=h(e^{-i\omega})\), \(\sum h_j e^{i\omega j}=h(e^{i\omega})\) and the spectrum \(s_x(\omega)\). \(\blacksquare\)

Definition 9.9 (Blaschke factor) The Blaschke factor \(B_\lambda(z)\) is defined by $$B_\lambda(z)=\frac{z-\lambda}{1-\lambda z}=-\lambda\frac{1-\lambda^{-1}z}{1-\lambda z}$$ with lag-operator form $$B_\lambda(L)=-\lambda\frac{1-\lambda^{-1}L}{1-\lambda L}$$ It flips the root \(\lambda\) to \(\lambda^{-1}\) (swapping inside and outside of the unit circle).

Definition 9.10 (Fundamental representation of MA(\(n\))) Consider MA(\(n\)): \(y_t=\theta_0(1-\lambda_1 L)(1-\lambda_2 L)\cdots(1-\lambda_n L)\epsilon_t\), where \(|\lambda_1|>|\lambda_2|>\dots>|\lambda_r|>1>|\lambda_{r+1}|>\dots>|\lambda_m|\). Its fundamental representation is $$y_t=C(L)u_t$$ where $$C(L)=\frac{B_{\lambda_1}(L)\cdots B_{\lambda_r}(L)}{(-\lambda_1)\cdots(-\lambda_r)\,\theta_0}\theta(L)=\Big(1-\tfrac1{\lambda_1}L\Big)\Big(1-\tfrac1{\lambda_2}L\Big)\cdots\Big(1-\tfrac1{\lambda_r}L\Big)(1-\lambda_{r+1}L)\cdots(1-\lambda_n L)$$ i.e. flip all the outside-the-circle roots \(\lambda_1,\dots,\lambda_r\) to inside, giving an invertible (fundamental) representation. Note also $$\operatorname{Var}(u_t)=(\lambda_1\lambda_2\cdots\lambda_r\theta_0)^2\operatorname{Var}(\epsilon_t)$$

Remark 9.19 When discussing Blaschke factors we implicitly require roots with \(|\lambda_j|\ne1\); otherwise flipping a root gives \(|\lambda_j^{-1}|=1\), still on the unit circle, corresponding to a persistent but non-exploding series. This is the knife-edge case between stationary and exploding: the series never returns to its original level, but it also does not diverge faster than a unit-root process.

Definition 9.11 (Unit root) An AR(\(m\)) process whose characteristic polynomial \(p(\lambda)\) has all roots \(\lambda_i\) (\(i=1,\dots,m\)) with modulus less than or equal to 1, and at least one root exactly equal to 1, is called integrated, or said to have a unit root.

Definition 9.12 (Integrated of order \(r\)) Suppose the roots \(\lambda_i\) (\(i=1,\dots,m\)) of the AR(\(m\)) characteristic polynomial \(p(\lambda)\) all have modulus less than or equal to 1. Let \(r\) be the number of roots exactly equal to 1; then the process is integrated of order \(r\), written \(I(r)\).

Remark 9.20 For an AR(\(m\)) process, the following two statements are equivalent: (i) the process is covariance stationary; (ii) the process is \(I(0)\). They are equivalent because, by definition, an \(I(0)\) process has all roots of the characteristic polynomial smaller than 1.

Definition 9.13 (Difference operator) The difference operator \(\Delta\) is defined by $$\Delta y_t=y_t-y_{t-1}=(1-L)y_t$$ Multiple differencing is indicated by powers: \(\Delta^r y_t=\Delta(\Delta(\cdots\Delta y_t))\). For example $$\Delta^2 y_t=\Delta(\Delta y_t)=\Delta(y_t-y_{t-1})=(y_t-y_{t-1})-(y_{t-1}-y_{t-2})=y_t-2y_{t-1}+y_{t-2}$$

Proposition 9.3 Let \(y_t\) be an \(I(r)\) AR(\(m\)) process. Then \(\Delta^r y_t\) is a covariance-stationary AR(\(m-r\)) process.

Proof Generally, factor the AR(\(m\)) characteristic polynomial and separate out the \(r\) unit roots: $$y_t=\frac1{\underbrace{(1-L)\cdots(1-L)}_{r\text{ unit roots}}\cdot(1-\lambda_{r+1}L)\cdots(1-\lambda_m L)}\cdot\epsilon_t$$ with \(|\lambda_{r+1}|,\dots,|\lambda_m|<1\). Hence $$\Delta^r y_t=(1-L)^r y_t=\frac1{(1-\lambda_{r+1}L)\cdots(1-\lambda_m L)}\cdot\epsilon_t$$ The right side is covariance stationary (all remaining roots inside the circle), i.e. \(\Delta^r y_t\) is a covariance-stationary AR(\(m-r\)). \(\blacksquare\)