4. Euler Equations, Transversality for Dynamic Problems

Chapter theme: Euler equations, transversality conditions and Hamiltonian for dynamic problems. §4.1 Discrete time: the state-sequence set-up \(V^\star(x_0)=\max\sum\beta^t F(x_t,x_{t+1})\) (states \(X\), correspondence \(\Gamma\), period-return \(F\), discount \(\beta\), value function \(V^\star\)) and the control-state set-up (controls \(U\), law of motion \(g\), \(F(x,y)=\max_u\{h(x,u):y=g(x,u)\}\)); Definition 4.1 Euler equation (EE) \(F_y(x_t,x_{t+1})+\beta F_x(x_{t+1},x_{t+2})=0\), Definition 4.2 transversality condition (TC) \(\lim_t\beta^t F_x(x_t,x_{t+1})\cdot x_t=0\); Proposition 4.1 EE+TC necessary & sufficient under regularity (concavity proof); Definition 4.3 steady state \(F_y(\bar x,\bar x)+\beta F_x(\bar x,\bar x)=0\); EE pins down \(\psi\) via lag 1/lag 2, Fact 4.1 uniqueness, Definition 4.4 shooting algorithm. §4.2 Continuous time: the state-sequence set-up \(\int e^{-\rho t}F(x,\dot x)dt\); Definition 4.5 EE \(F_x+\rho F_{\dot x}=F_{\dot x x}\dot x+F_{\dot x\dot x}\ddot x\) (4.3), Definition 4.6 TC \(\lim e^{-\rho T}F_{\dot x}x=0\); Proposition 4.2 necessary & sufficient (concavity + integration by parts); Definition 4.7 steady state \(F_x(\bar x,0)+\rho F_{\dot x}(\bar x,0)=0\); control-state set-up + the maximum principle (Hamiltonian): Definition 4.8 \(H=h+\lambda g\), Proposition 4.3 \(H_u=0\), \(\dot\lambda=\rho\lambda-H_x\), \(\dot x=g\), TC \(\lim e^{-\rho T}\lambda x=0\) (Remark 4.4 co-state interpretation, heuristic Lagrangian proof). §4.2.6 relationship between Hamiltonian and EE (\(\lambda=-F_{\dot x}\) ⟹ back to EE). §4.3 summary of both set-ups in discrete and continuous time.

Definition 4.1 (Euler Equation - discrete time) The path \(\{x_{t+1}\}_{t=0}^\infty\) satisfies EE if $$F_y(x_t,x_{t+1})+\beta F_x(x_{t+1},x_{t+2})=0,\quad\text{for }\forall t\ge0$$

Definition 4.2 (Transversality Condition - discrete time) The path \(\{x_{t+1}\}_{t=0}^\infty\) satisfies TC if $$\lim_{t\to\infty}\beta^t F_x(x_t,x_{t+1})\cdot x_t=0$$

Proposition 4.1 Under regularity conditions, EE and TC are necessary and sufficient for \(\{x_{t+1}\}_{t=0}^\infty\) to be optimal, i.e. the solution to the state sequence problem.

Remark 4.1 Since \(x\) could potentially be a vector, the EE must hold for each element of \(x\). And in the definition of TC, the dot means inner product.

Remark 4.2 Actually, EE is just the f.o.c. of \(x_{t+1}\) for \(\forall t\ge0\). TC means that the return in the too far away future is not important. EE requires maximization in each period, which is a concept of local allocation maximizing. Note that even the EE is satisfied, we could end up obtaining a sequence \(\{x_{t+1}\}_{t=0}^\infty\) such that each period has a lower period-return and resources are accumulated to infinity in the future that will always be pushed further and never realizes. Such sequence is clearly not optimal, so we need TC to avoid being trapped in this fake maximizing sequence, which makes TC a concept of overall allocation maximizing.

Proof (Proposition 4.1, under regularity conditions) Regularity conditions: \(F(x,y)\) is concave in \((x,y)\); \(F_x(x^\star_t,x^\star_{t+1})\ge0\) where \(\{x^\star_{t+1}\}_{t=0}^\infty\) is the optimal path; \(F\in C^1\).

Sufficiency. Since \(f\) is concave, $$f(x)-f(x_0)\le f'(x_0)(x-x_0),\quad\text{for }\forall x\tag{4.1}$$ (Note that if \(x\in\mathbb R^m\) and \(f:\mathbb R^m\to\mathbb R\), then \(f'(x_0)\) is a \(1\times m\) partial vector evaluated at \(x_0\).) The goal is to show that if \(\{x^\star_{t+1}\}_{t=0}^\infty\) satisfies EE and TC, then it is the optimal path. Take an arbitrary path \(\{x_{t+1}\}_{t=0}^\infty\) with \(x_0=x^\star_0\), and we want to show that \(\lim_{T\to\infty}\sum_{t=0}^T\beta^t[F(x_t,x_{t+1})-F(x^\star_t,x^\star_{t+1})]\le0\). Using (4.1), $$F(x_t,x_{t+1})-F(x^\star_t,x^\star_{t+1})\le F_x(x^\star_t,x^\star_{t+1})(x_t-x^\star_t)+F_y(x^\star_t,x^\star_{t+1})(x_{t+1}-x^\star_{t+1})$$ Sum and regroup: note \(x_0=x^\star_0\) makes the first term zero, each square bracket containing EE \([F_y(x^\star_t,x^\star_{t+1})+\beta F_x(x^\star_{t+1},x^\star_{t+2})]\) is zero, so only the last term stays: $$\begin{aligned}\lim_{T\to\infty}\sum_{t=0}^T\beta^t[\cdots]&\le\lim_{T\to\infty}\beta^T F_y(x^\star_T,x^\star_{T+1})(x_{T+1}-x^\star_{T+1})\\&=\lim_{T\to\infty}-\beta^{T+1}F_x(x^\star_{T+1},x^\star_{T+2})(x_{T+1}-x^\star_{T+1})\\&\le\lim_{T\to\infty}\beta^{T+1}F_x(x^\star_{T+1},x^\star_{T+2})x^\star_{T+1}=0\quad\because\text{TC}\end{aligned}$$ where the second last line uses EE (\(F_y(x^\star_T,x^\star_{T+1})=-\beta F_x(x^\star_{T+1},x^\star_{T+2})\)), and the last drops the non-negative term using \(F_x\ge0\) and \(x_{T+1}\ge0\) and reaches zero by TC.

Necessity. Add a variation around the optimal path \(\{x^\star_{t+1}\}_{t=0}^\infty\). Consider \(\{x_{t+1}(\alpha,\varepsilon)\}_{t=0}^\infty\) where \(x_t(\alpha,\varepsilon)=x^\star_t+\alpha\varepsilon_t\) for \(\forall t\ge0\), \(\alpha\in\mathbb R\), \(\varepsilon=\{\varepsilon_t\}_{t=0}^\infty\) with \(\varepsilon_t\in\mathbb R^m\), and set \(\varepsilon_0=0\) to make the sequences comparable. Define $$V^\star(x_0)=v(0)=\lim_{T\to\infty}\sum_{t=0}^T\beta^t F(x_t(0,\varepsilon),x_{t+1}(0,\varepsilon))=\lim_{T\to\infty}\sum_{t=0}^T\beta^t F(x^\star_t,x^\star_{t+1})$$ $$v(\alpha)=\lim_{T\to\infty}\sum_{t=0}^T\beta^t F(x_t(\alpha,\varepsilon),x_{t+1}(\alpha,\varepsilon))$$ for any \(\alpha,\varepsilon\) such that \(x_{t+1}(\alpha,\varepsilon)\in\Gamma(x_t(\alpha,\varepsilon))\) (the variation path is feasible). \(v(0)\) is the maximum, so \(\alpha=0\) maximizes \(v\), and the f.o.c. \(\frac{\partial v(0)}{\partial\alpha}=0\). Suppose differentiation and taking limit are interchangeable; after rearranging, $$0=\frac{\partial v(0)}{\partial\alpha}=\lim_{T\to\infty}\sum_{t=0}^T\beta^t\left[F_y(x^\star_t,x^\star_{t+1})+\beta F_x(x^\star_{t+1},x^\star_{t+2})\right]\varepsilon_{t+1}+\beta^T F_y(x^\star_T,x^\star_{T+1})\varepsilon_{T+1}\tag{4.2}$$ To make sure any deviation from the optimal path in any period is sub-optimal, we need (4.2) to hold for \(\varepsilon_{t+1}\ne0\) for any single \(t+1\), so the square bracket must be zero — which is EE. Then $$0=\frac{\partial v(0)}{\partial\alpha}=\lim_{T\to\infty}\beta^T F_y(x^\star_T,x^\star_{T+1})\varepsilon_{T+1}$$ Take \(\varepsilon_{T+1}=x^\star_{T+1}\) and substitute \(T\) for \(T+1\): $$\lim_{T\to\infty}\beta^T F_x(x^\star_T,x^\star_{T+1})x^\star_T=0$$ which is exactly TC. \(\blacksquare\)

Definition 4.3 (Steady state) \(\bar x\) is a steady state if it solves $$F_y(\bar x,\bar x)+\beta F_x(\bar x,\bar x)=0$$

Remark 4.3 The reason why \(\bar x\) is called the steady state is that after reaching \(x_s=\bar x\), the optimal path thereafter is to stay put, i.e. \(x_t=\bar x\) for \(\forall t\ge s\). Why is staying put optimal? By definition, \(\bar x\) staying put satisfies EE. And it also satisfies TC since \(F_x(\bar x,\bar x)\bar x\) is a fixed number and thus \(\lim\beta^t F_x(\bar x,\bar x)\bar x=0\). Notice such argument depends on the assumption that \(F(x,y)\) is concave in \((x,y)\), since the sufficiency of EE and TC depends on the concavity of \(F\).

Fact 4.1 If \(X\), \(F\), \(\Gamma\) satisfies the following convexity conditions, then the dynamic problem has at most one solution: \(X\subseteq\mathbb R^m\); \(F(x,y)\) is \(C^1\), \(F(x,y)\) is strictly concave in \((x,y)\), \(F(x,y)\) is strictly increasing in \(x\), and \(\beta\in(0,1)\); \(\Gamma\) is convex and increasing in \(x\), i.e. \(x^\star>x\Rightarrow\Gamma(x)\subseteq\Gamma(x^\star)\).

Definition 4.4 (Shooting algorithm) Given \(x_0\), arbitrarily select an \(x_1\), then use EE to generate a sequence \(\{x_t\}_{t=2}^\infty\). Check if this sequence satisfies TC. If yes, by the sufficiency of EE and TC, this sequence is optimal. If no, try another \(x_1\).

Definition 4.5 (Euler Equation - continuous time) The path \(\{\dot x(t)\}_{t=0}^\infty\) satisfies EE if $$F_x(x(t),\dot x(t))+\rho F_{\dot x}(x(t),\dot x(t))=F_{\dot x x}(x(t),\dot x(t))\dot x(t)+F_{\dot x\dot x}(x(t),\dot x(t))\ddot x(t)\tag{4.3}$$ for \(\forall t\ge0\).

Definition 4.6 (Transversality Condition - continuous time) The path \(\{\dot x(t)\}_{t=0}^\infty\) satisfies TC if $$\lim_{T\to\infty}e^{-\rho T}F_{\dot x}(x(T),\dot x(T))\cdot x(T)=0$$

Proposition 4.2 Under regularity conditions, EE and TC are necessary and sufficient for \(\{\dot x(t)\}_{t=0}^\infty\) to be optimal, i.e. the solution to the continuous state sequence problem.

Proof (Proposition 4.2, under regularity conditions) Regularity conditions: \(F(x,\dot x)\) is concave in \((x,\dot x)\) (for sufficiency); \(F_{\dot x}\le0\) (for sufficiency); \(\text{Gr}(\Gamma)\) is convex; optimal path is interior; \(F\in C^1\) (for necessity); \(V(\alpha)\) is differentiable (for necessity).

Sufficiency. We want to show \(\lim_{T\to\infty}\int_0^T e^{-\rho t}(F(x(t),\dot x(t))-F(x^\star(t),\dot x^\star(t)))\,dt\le0\). Since \(F\) is concave, $$F(x(t),\dot x(t))-F(x^\star(t),\dot x^\star(t))\le F_x(x^\star,\dot x^\star)(x-x^\star)+F_{\dot x}(x^\star,\dot x^\star)(\dot x-\dot x^\star)$$ So \(\lim\int\le\underbrace{\lim\int e^{-\rho t}F_x(x^\star,\dot x^\star)(x-x^\star)dt}_{\text{Part A}}+\underbrace{\lim\int e^{-\rho t}F_{\dot x}(x^\star,\dot x^\star)(\dot x-\dot x^\star)dt}_{\text{Part B}}\). Apply integration by parts to Part B: $$\begin{aligned}\text{Part B}&=\left[e^{-\rho t}F_{\dot x}(x^\star,\dot x^\star)(x-x^\star)\right]_0^T-\int_0^T e^{-\rho t}\underbrace{\left[F_{\dot x x}\dot x^\star+F_{\dot x\dot x}\ddot x^\star-\rho F_{\dot x}-F_x\right]}_{=0\text{ by EE}}(x-x^\star)dt\\&\quad-\int_0^T e^{-\rho t}(x-x^\star)F_x\,dt\\&=\underbrace{\lim_{T\to\infty}e^{-\rho T}F_{\dot x}(x^\star(T),\dot x^\star(T))x(T)}_{\le0}-\underbrace{\lim_{T\to\infty}e^{-\rho T}F_{\dot x}(x^\star(T),\dot x^\star(T))x^\star(T)}_{=0\ \because\text{TC}}\\&\quad-\lim_{T\to\infty}\int_0^T e^{-\rho t}(x-x^\star)F_x\,dt\\&\le-\text{Part A}\end{aligned}$$ So \(\text{Part A}+\text{Part B}\le0\).

Necessity. Variation path \(x_{\alpha,\varepsilon}(t)=x(t)+\alpha\varepsilon(t)\), \(\alpha\in\mathbb R\), \(\varepsilon(t):\mathbb R_+\to\mathbb R^m\) differentiable with \(\varepsilon(0)=0\), and \(\{x(t)\}_{t=0}^\infty\) optimal. $$v(\alpha)=\lim_{T\to\infty}\int_0^T e^{-\rho t}F(x_{\alpha,\varepsilon}(t),\dot x_{\alpha,\varepsilon}(t))\,dt=\lim_{T\to\infty}\int_0^T e^{-\rho t}F(x(t)+\alpha\varepsilon(t),\dot x(t)+\alpha\dot\varepsilon(t))\,dt$$ The variational path is feasible \(\dot x_{\alpha,\varepsilon}(t)\in\Gamma(x_{\alpha,\varepsilon}(t))\). \(v(0)\ge v(\alpha)\), \(v\) is differentiable, f.o.c. \(\frac{\partial v(0)}{\partial\alpha}=0\): $$0=\frac{\partial v(0)}{\partial\alpha}=\underbrace{\lim_{T\to\infty}\int_0^T e^{-\rho t}F_x(x,\dot x)\varepsilon\,dt}_{\text{Part A}}+\underbrace{\lim_{T\to\infty}\int_0^T e^{-\rho t}F_{\dot x}(x,\dot x)\dot\varepsilon\,dt}_{\text{Part B}}$$ Applying integration by parts to Part B and combining with Part A, $$0=\frac{\partial v(0)}{\partial\alpha}=\underbrace{\lim_{T\to\infty}e^{-\rho T}F_{\dot x}(x(T),\dot x(T))\varepsilon(T)}_{(4.4)}+\underbrace{\lim_{T\to\infty}\int_0^T e^{-\rho t}\left[F_x+\rho F_{\dot x}-F_{\dot x x}\dot x-F_{\dot x\dot x}\ddot x\right]\varepsilon\,dt}_{(4.5)}$$ To make sure deviation in any period is sub-optimal, we need the integral in (4.5) to be zero for any \(\varepsilon(t)\ne0\) — which is EE. Also consider the first term (4.4), for the particular case \(\varepsilon(T)=x(T)\), this term should be zero — which is TC. \(\blacksquare\)

Definition 4.7 (Steady state) \(\bar x\) is a steady state if it solves $$F_x(\bar x,0)+\rho F_{\dot x}(\bar x,0)=0$$

Definition 4.8 (co-state and Hamiltonian) Let \(\lambda\) be a vector on \(\mathbb R^m\) called co-state variable, and let \(H\) be the Hamiltonian function. They are defined as $$H(x,u,\lambda)=h(x,u)+\lambda g(x,u)$$ where \(h\) and \(g\) are the same functions as in subsection 4.2.4.

Proposition 4.3 Under regularity conditions, the following conditions are necessary and sufficient for the path of \(\{x(t)\}_{t=0}^\infty\) and \(\{u(t)\}_{t=0}^\infty\) to be optimal: $$H_u(x(t),u(t),\lambda(t))=0$$ $$\dot\lambda(t)=\rho\lambda(t)-H_x(x(t),u(t),\lambda(t))$$ $$\dot x(t)=g(x,u)$$ for \(\forall t\ge0\). The state variable \(x\) has an initial value of \(x_0\), and the co-state variable \(\lambda\) has a boundary condition (TC): $$\lim_{T\to\infty}e^{-\rho T}\lambda(T)x(T)=0$$

Remark 4.4 (interpretations of the Hamiltonian) - The co-state variable \(\lambda(t)\) can be interpreted as the present value of one unit increase in the state variable at period \(t\). The law of motion of co-state simply means that the present value of one unit increase will grow as time flows, but it will also lose the current period return that one unit increase counted as past thereafter. So the net of the two effects will result in the change in the co-state variable. - The second and the third equation are just the law of motion of co-state and state variables by their nature and definition. The only optimization part comes from the f.o.c. of the control variable, which is the first equation. It gives us the optimal control that balances the instantaneous return measured by \(h\) and discounted future returns measured by \(\lambda g\). - \(H(x,u,\lambda)\) is the instantaneous return plus the discounted future return for one unit increase in the state variable. The design of this function requires the f.o.c. w.r.t. control variables to take both current return and future returns into consideration, which determines the optimal control.

Heuristic proof We can form the Lagrangian for the continuous control-state sequence set-up to derive the Hamiltonian conditions. We will use \(e^{-\rho t}\lambda(t)\) as the Lagrangian multiplier of \(\dot x(t)=g(x(t),u(t))\) to transfer everything into the present value of \(t=0\) to make them comparable: $$\mathcal L(x,u,\lambda)=\lim_{T\to\infty}\left(\int_0^T e^{-\rho t}h(x(t),u(t))\,dt+\int_0^T e^{-\rho t}\lambda(t)[g(x(t),u(t))-\dot x(t)]\,dt\right)$$ First, do integral by parts on \(\int_0^T e^{-\rho t}\lambda(t)\dot x(t)\,dt\): $$\int_0^T e^{-\rho t}\lambda(t)\dot x(t)\,dt=\left[e^{-\rho t}\lambda(t)x(t)\right]_0^T-\int_0^T\left[-\rho e^{-\rho t}\lambda(t)+e^{-\rho t}\dot\lambda(t)\right]x(t)\,dt$$ Plug in and rearrange the Lagrangian: $$\mathcal L(x,u,\lambda)=\lim_{T\to\infty}\int_0^T e^{-\rho t}h\,dt+\lim_{T\to\infty}\int_0^T e^{-\rho t}\left[\lambda g(x,u)-\rho\lambda(t)x(t)+\dot\lambda(t)x(t)\right]dt$$ Take derivative w.r.t. \(x(t)\): $$[x(t)]:\quad\frac{d\mathcal L}{dx(t)}=e^{-\rho t}\left[h_x+\lambda g_x-\rho\lambda+\dot\lambda\right]=0\Rightarrow\dot\lambda=\rho\lambda-h_x-\lambda g_x=\rho\lambda-H_x\tag{4.6}$$ Take derivative w.r.t. \(u(t)\): $$[u(t)]:\quad\frac{d\mathcal L}{du(t)}=e^{-\rho t}\left[h_u+\lambda g_u\right]=0\Rightarrow h_u+\lambda g_u=H_u=0\tag{4.7}$$ Take derivative w.r.t. \(\lambda(t)\): $$[\lambda(t)]:\quad\frac{d\mathcal L}{d\lambda(t)}=e^{-\rho t}\left[g(x,u)-\dot x\right]=0\Rightarrow\dot x=g(x,u)$$ \(\blacksquare\)