14. How to do Good Empirical Work

1. Defining the target parameters (sometimes called the parameters of interest). - A target parameter is always defined by a counterfactual that we are interested in. - Thinking about counterfactuals requires doing a thought experiment: e.g. "what would happen to unemployment if the government changes the minimum wage?", "what would happen to prices if two firms merge?" - This step is purely a thought experiment, requiring neither data nor experiment — which is contrary to the slogan "no causation without manipulation". - In this step, the key is to precisely specify the counterfactual question and unambiguously define the parameter we want to explore in the following steps.

2. Identification of the target parameter. - The target parameter is generally a function of both observables (e.g. race and gender) and unobservables (e.g. ability, intelligence, and the potential outcome of those not treated were they actually treated). - The question of identification is: what can we learn about the target parameter from observable data given that we have the data of the population (i.e. not restricted by limited sampling). - A model or theory is a set of assumptions explicitly made to help with identification. - The identification step maps assumptions (model) and population data to the information about the target parameter: - A parameter is identified if, under the stated assumptions, alternative values of the target parameter imply different distributions of the observable data of the population. - Identification has a binary property: a yes-or-no question, i.e. either target parameters are identified or not. - This step is just theoretical or hypothetical, and is a prerequisite for step 3: in reality, we never have access to population data in general; identification basically asks the simple question: can you recover the target parameter when you know the population distribution? If the answer to the identification question is no, then you also won't be able to learn about the target parameter with a finite sample that we have to deal with.

3. Statistical inference. - In practice, we only see a finite sample of data on the observables. From this we know the sample distribution. However, we don't know the population distribution of the data. - Statistical inference is using the sample to learn about the population. - It's useful to separate identification from statistical inference, and always do identification before statistical inference.

The overall flow: $$\text{sample}\ \xrightarrow{\text{statistical inference}}\ \text{population}\ \xrightarrow{\text{identification}}\ \text{unobserved parameters}$$

Formal models should be very precise in their target parameter definition, identification, and inference. But different researchers might use very different notation and model set-up for the same problem. This section uses the following two models as examples: - Potential outcome model: the Neyman–Fisher–Quandt–Rubin model. - Economic choice models (latent variable models): a general set of models; we will in particular focus on the Roy model.

Set-up. This model offers one way of modeling the world in contrast to the Roy model, which is an alternative but equivalent view of the same problem. - \(\mathcal D\) is a mutually exclusive and exhaustive set of states. E.g. in a specific setting of the education decision model, \(\mathcal D=\{0,1\}\), getting college means \(d=1\in\mathcal D\) whereas not getting college means \(d=0\in\mathcal D\). - For each \(d\in\mathcal D\), \(Y_d\) is the potential outcome — what would have happened if the state were endogenously set \(d\). In fact we can only observe the realized one outcome, which is the fundamental challenge to identification. In the education decision model, the thought experiment is what happens when \(d=1\) as opposed to \(d=0\). - We observe the actual state, a random variable \(D\in\mathcal D\) and a corresponding outcome \(Y\), which satisfies $$Y=\sum_{d\in\mathcal D}Y_d\mathbf 1\{D=d\}=Y_D$$ when \(D\in\mathcal D\) is observed, only \(Y=Y_D\) is revealed, but \(Y_d\) for \(d\ne D\) is unobserved.

Let's focus on the simple example of the education decision model. The switching regression is $$Y=DY_1+(1-D)Y_0\tag{14.1}$$ \(D\in\{0,1\}\) is a choice: \(D=1\) going to college, \(D=0\) not; \(Y_1\) is the potential outcome when \(D=1\), \(Y_0\) the potential outcome when \(D=0\).

OLS regression. Without further restrictions \(Y_1-Y_0\) may vary across individuals in general, so write an OLS regression with individual subscript \(i\): $$Y_i=\alpha_i+\beta^{\text{OLS}}\cdot D_i+\varepsilon_i\tag{14.2}$$ Rearrange the switching regression (14.1) to match (14.2): $$Y_i=D_iY_{i,1}+(1-D_i)Y_{i,0}\Rightarrow Y_i=\underbrace{\mathbb E[Y_{i,0}]}_{\alpha_i}+\underbrace{(Y_{i,1}-Y_{i,0})}_{\beta_i}\cdot D_i+\underbrace{Y_{i,0}-\mathbb E[Y_{i,0}]}_{\varepsilon_i}$$ or \(Y_i=\underbrace{(Y_{i,1}-Y_{i,0})}_{\beta_i}\cdot D_i+Y_{i,0}\), where \(\beta_i\) is the individual treatment effect. The explicit expression for \(\beta^{\text{OLS}}\) (derivation below) satisfies $$\beta^{\text{OLS}}=\mathbb E[Y\mid D=1]-\mathbb E[Y\mid D=0]\tag{14.4}$$

Case 1: \((Y_0,Y_1)\perp D\) (independence). Then the \(\beta^{\text{OLS}}\) from OLS is the population average of \(\beta_i=Y_{i,1}-Y_{i,0}\): $$\begin{aligned}\beta^{\text{OLS}}&=\mathbb E[Y\mid D=1]-\mathbb E[Y\mid D=0]=\mathbb E[DY_1+(1-D)Y_0\mid D=1]-\mathbb E[DY_1+(1-D)Y_0\mid D=0]\\&=\mathbb E[Y_1\mid D=1]-\mathbb E[Y_0\mid D=0]\overset{\perp}{=}\mathbb E[Y_1]-\mathbb E[Y_0]=\underbrace{\mathbb E[Y_1-Y_0]}_{\text{ATE}}\end{aligned}$$ i.e. \(\text{ATE}=\mathbb E[Y_1-Y_0]=\mathbb E[\beta_i]\).

Case 2: \((Y_0,Y_1)\not\perp D\) (non-independence). The model says nothing about why individuals take treatment (go to college), but it does not preclude the probability that \(D\) could be dependent on \((Y_0,Y_1)\): - If \(D\) is dependent on \(Y_1-Y_0\), say \(D=\mathbf 1\{Y_1-Y_0>0\}=\mathbf 1\{Y_1>Y_0\}\) (or more generally \(\operatorname{Cov}(Y_1-Y_0,D)>0\)), then it indicates selection on the gains (only people receiving high gains from education would go to college). - If \(D\) is dependent on \(Y_0\), say \(D=\mathbf 1\{Y_0\ge C\}\) (\(C\) some critical value, or more generally \(\operatorname{Cov}(Y_0,D)>0\)), then it indicates selection bias (only the people with high enough \(Y_0\) could be treated, i.e. only smart enough people can go to college). - Sometimes we could have selection on the gains or selection bias or both.

Instrumental variable. Introduce an instrument \(Z\): $$D=ZD_1+(1-Z)D_0$$ where \(Z\in\{0,1\}\) and \(D_z\) is the value of realized treatment \(D\) if \(Z\) was endogenously set \(z\). This is an example of a binary instrument, and we can think of the instrument as partitioning agents into four different groups as discussed in Case 2 of §5.6: - Always takers: \(D_1=D_0=1\), i.e. those who always take treatment regardless of the value of the instrument. - Compliers: \(D_1>D_0\), i.e. those who take treatment when \(Z=1\) and don't take treatment when \(Z=0\). - Never takers: \(D_1=D_0=0\), i.e. those who never take treatment regardless of the value of the instrument. - Defiers (ruled out by the monotonicity assumption): \(D_1

As shown in §5.6, if the instrument \(Z\) is properly introduced (i.e. satisfying exogeneity, relevance and monotonicity), then \(\beta^{\text{IV}}\) is the local average treatment effect (LATE), which is the ATE of the compliers group.

In the potential outcome model, which has the relatively easier setting, the key is that we don't impose a lot of structure on how the individuals make decisions on whether or not to take treatment. We do, however, need to take a strong stand on counterfactuals: depending on the question you ask, you will get different target parameters.

An alternative to potential outcome notation is latent variable notation, in the outcome equation, the choice equation, or both.

Latent variable notation to describe outcomes. Many times latent variable models introduce an unobserved variable \(V\) into the outcome equation and write $$Y=g(D,V)$$ where \(g\) is a function of the unobserved variable \(V\) and the choice to make \(D\). A causal interpretation of this model is implicitly saying: \(Y_d=g(d,V)\) for every \(d\in\mathcal D\). It is also typical to make some restrictions on \(g\), e.g. \(g\) being Cobb–Douglas.

Latent variable notation to describe choices. The leading case is binary \(D\) with a separable latent variable choice equation $$D=\mathbf 1\Big\{\underbrace{U}_{\text{latent variable}}\le\underbrace{\nu(W)}_{\text{unknown function}}\Big\}$$ where: \(W\equiv(X,Z)\) are observables with \(Z\) being the instruments; \(U\) can intuitively be understood as the unwillingness to take the treatment, continuously distributed and normalized to be uniform on $[0,1]$. - Note that the assumption \(U\sim\text{Unif}[0,1]\) is without loss of generality, because even in the general case where \(U\) follows some other distribution, we still have \(U^\star=F_U(U)\), which is the CDF of \(U\), that follows the uniform distribution on $[0,1]$.

Define \(\nu^\star(W)\equiv F_U(\nu(W))\); then $$D=\mathbf 1\{U\le\nu(W)\}\overset{F_U\text{ incr.}}{=}\mathbf 1\{F_U(U)\le F_U(\nu(W))\}=\mathbf 1\{U^\star\le\nu^\star(W)\}$$ i.e. we can always transform a non-uniformly distributed \(U\) to a uniformly distributed \(U^\star\) on $[0,1]$.

\(p(W)\) is called the propensity score.

A special case of the latent variable model is a combination of the separable latent variable choice equation defined above plus the potential outcome equation, i.e. $$Y=Y_1D+Y_0(1-D),\qquad D_z=\mathbf 1\Big\{U\le\nu\big(\underbrace{W}_{=(X,Z)}\big)\Big\}$$ A common version of the Roy Model is $$\begin{aligned}Y_0&=X'\beta_0+V_0\\Y_1&=X'\beta_1+V_1\\D&=\mathbf 1\{U\le W'\gamma\}\quad(\text{choice or selection equation})\end{aligned}$$ where \((V_0,V_1,U)\) are unobservable variables and \(W\equiv(X,Z)\) are observable variables. Notice that the difference in potential outcomes is a function of both observed and unobserved heterogeneity: $$Y_1-Y_0=\underbrace{X'(\beta_1-\beta_0)}_{\text{observed}}+\underbrace{V_1-V_0}_{\text{unobserved}}$$ When \(U\) and \((V_0,V_1)\) are dependent, the selection decision \(D\) is dependent on \((V_0,V_1)\), so we call it selection on unobservables. And notice that $$\begin{aligned}Y&=DY_1+(1-D)Y_0=(Y_1-Y_0)\cdot D+Y_0=[X'(\beta_1-\beta_0)+V_1-V_0]\cdot D+X'\beta_0+V_0\\&=X'(\beta_1-\beta_0)\cdot D+(V_1-V_0)\cdot D+X'\beta_0+V_0\end{aligned}$$ breaks down the effect (coefficient) of treatment \(D\) into two parts: \(X'(\beta_1-\beta_0)\) through observable \(X\), and \((V_1-V_0)\) through unobservable \((V_0,V_1)\).

Notice that there are some buried assumptions in the Roy model. In particular: 1. No equilibrium effects. To understand what this means, consider a policy change of a decrease in tuition for schooling where tuition is one observable variable in \(X\). If the decrease in tuition itself was large enough, the skill premium for additional schooling would decrease, which decreases \(Y_1-Y_0\). This decrease in the skill premium is an equilibrium effect. So for this example, when we say no equilibrium effects, we're essentially saying that the policy change is small enough not to affect the skill premium, which means \(X\) only affects the value of \(D\), not the value of \(Y_0\) and \(Y_1\). 2. No heterogeneity in coefficients among agents. The values of \(\beta_0,\beta_1\) and \(\gamma\) don't vary among agents.

There is unobserved treatment heterogeneity if and only if non-constant MTE. Many quantities can be written as a weighted average of the MTE: - ATE is the unweighted average of the MTE: $$\text{ATE}=\mathbb E[\mathbb E[Y_1-Y_0\mid U=u]]=\int_0^1\text{MTE}(u)\times\underbrace{1}_{\omega_{\text{ATE}}(u)}\,du$$ where \(\omega_{\text{ATE}}(u)=1\) comes from \(U\sim\text{Unif}[0,1]\). - ATT (average treatment effect on the treated) is the weighted average of the MTE: $$\text{ATT}=\mathbb E[Y_1-Y_0\mid D=1]=\int_0^1\text{MTE}(u)\underbrace{\frac{\mathbb P(p(Z)\ge u)}{\mathbb P(D=1)}}_{\omega_{\text{ATT}}(u)}\,du$$

Intuition: \(\mathbb P(p(Z)\ge u)\) is the fraction of people with \(U=u\) taking treatment, and the denominator \(\mathbb P(D=1)\) normalizes it so the weights add up to 1. \(\omega_{\text{ATT}}(u)\) is the weight of treated people with \(U=u\) in the group of all treated people, and we're upweighting people with lower \(u\) (lower unwillingness to take treatment).

• ATU (average treatment effect on the untreated) is also a weighted average of the MTE: $$\text{ATU}=\mathbb E[Y_1-Y_0\mid D=0]=\int_0^1\text{MTE}(u)\underbrace{\frac{\mathbb P(p(Z)

• LATE (local average treatment effect for the compliers) is also a weighted average of the MTE. Recall (5.42) in §5.6, which in the notation here is $$\text{LATE}=\frac{\mathbb E[Y\mid Z=z]-\mathbb E[Y\mid Z=z']}{\mathbb E[D\mid Z=z]-\mathbb E[D\mid Z=z']}\tag{14.5}$$

Define the Policy Relevant Treatment Effect (PRTE) as the aggregate effect on \(Y\) per net person of a change in the propensity score (or probability of taking treatment), both of which are caused by the introduction of a policy: $$\beta_{\text{PRTE}}\equiv\frac{\overbrace{\mathbb E[Y^\star]-\mathbb E[Y]}^{\text{mean effect}}}{\underbrace{\mathbb E[D^\star]-\mathbb E[D]}_{\text{per net person}}}\tag{14.6}$$ where \(Y,D\) are pre-policy values and \(Y^\star,D^\star\) are post-policy values. In other words, the policy first affects the proportion of people who take treatment by \(\mathbb E[D^\star]-\mathbb E[D]\) (i.e. affecting the incentive to take treatment); then the fraction of those net new people who switch to get treatment, times the multiplier \(\beta_{\text{PRTE}}\), gives us the overall effect \(\mathbb E[Y^\star]-\mathbb E[Y]\) on the average outcome due to the policy.

\(\beta_{\text{PRTE}}\) can also be expressed as a weighted average of the MTE: $$\beta_{\text{PRTE}}=\int_0^1\text{MTE}(u)\omega_{\text{PRTE}}(u)\,du$$ PRTE is the ATE of the group of compliers to the policy change, which is a more general version of LATE. So \(\omega_{\text{PRTE}}(u)\) is the fraction (w.r.t. the whole group) of compliers to a policy at each \(U=u\) divided by fraction of compliers in the whole group.

Consider the following contingency table where \(E\) means employment, \(N\) means unemployment, and the treatment is a training program:

Suppose we believe that \(P_{E\bullet}=0.6\) and \(P_{\bullet E}=0.6\); then \(P_{EE}=0.2\) is possible but \(P_{EE}=0.1\) is not possible, because \(P_{EE}=0.1\) implies \(P_{NE}=0.5\) and \(P_{EN}=0.5\), then the joint probabilities sum up to exceed 1. This simple example illustrates that when theory tells us only the marginal distribution (e.g. \(P_{E\bullet}=0.6\) and \(P_{\bullet E}=0.6\)) but not the joint distribution, we can at least put some bounds on some joint probabilities (e.g. \(0.2\le P_{EE}\le0.6\)). For another example, if a training program unambiguously increases your probability of getting a job, then \(P_{NE}\) must be zero. In conclusion, theory could give us additional restrictions on the joint distribution not implied by the experiment or data.

• Experiment is useful for Mechanism Separation researches. For example, peer effect has two possible channels (mechanisms) to affect people's behavior (e.g. financial investment decision):
• Social learning: information updating from observing what others do. E.g. others buy this stock, and I think their choices reveal some information telling me that the stock is good, so I also buy it because of the information collected through observation.
• Social utility: direct effect on the utility function. E.g. I buy this stock because all others buy and I want to have topics in common with the group.
• It is very hard to separate such mechanisms using only naturally collected data without carefully designing experiments.
• Always think about the internal validity and external validity.
• First, make sure the experiment has internal validity. Sometimes you might forget to add in some alternative explanations for an observation, then the referee and seminar audience will outsmart you by challenging or even destroying the credibility of the story (explanation) in the paper. So, always think about as many possible alternative explanations as possible, and then either control them during your experiment or discuss them by yourself using no less than 8 pages to show that you have already taken those factors into consideration and they are not ruining your experiment design.
• Then, be careful to draw any conclusions about the external validity. A lab experiment has more internal validity through more rigorous control, but is less "real" so has less external validity. A field experiment has more interference so less internal validity, but possibly more external validity. So take this trade-off into consideration and choose your experiment method accordingly.
• It's a good idea to run follow-up surveys, which might potentially give you more findings out of the results. But nowadays it has become a norm to pre-register your experiment, and some people really take it to the extreme. So don't forget to design those surveys ahead of time and include them in the pre-registration.
• Qualitative vs quantitative conclusion.
• If the intended result is qualitative, or directional, then less controls and more real might be the right choice, and external validity is somewhat more important.
• If the intended result is quantitative, which means high accuracy, then more controls is necessary even inevitably at the cost of less real, and internal validity is somewhat more important.