19. Learning from Data

Empirical research can be roughly divided into four styles: descriptive studies ("just the facts"), causal analysis (experiments / natural experiments / "treatment effects"), structural analysis, and calibration. The following table compares the four along several dimensions.

19.2.1 Set-up - A vector \(\mathbf W\) of observables, distributed according to \(g\in G\). - \(G\) is a subset of the set of all possible distributions of \(\mathbf W\), and \(g\) is a known distribution. - Note that the distribution is for the population, not the sample. There is no sampling involved in the step of identification. - \(\Theta\) is the set of all allowable models. - \(\theta\in\Theta\) is a model. - Parameter of interest is \(\rho=\pi(\theta)\). - \(\pi(\cdot)\) is a function that maps the model to the parameter of interest.

19.2.2 Definition of identification

Identification requires that given the observed distribution \(g\) of \(\mathbf W\), we have 1. \(\theta\in\Theta\); 2. \(g_\theta=g\).

So, denote the set of identified models by \(\Theta^\star\) such that $$\Theta^\star=\{\theta:\theta\in\Theta\text{ and }g_\theta=g\}$$ Denote the set of identified parameters of interest by \(\Pi^\star\) such that $$\Pi^\star=\{\pi(\theta):\theta\in\Theta^\star\}$$

19.2.3 Example: identification for linear regression - The vector of observables is \(\mathbf W=(Y,\mathbf X)\). - \(Y\in\mathbb R\); - \(\mathbf X\in\mathbb R^k\). - The set of all allowable models is $$\Theta=\left\{(\boldsymbol\beta,F):\mathbb E_F[\mathbf X u]=0,\ \mathbb E_F[\mathbf X\mathbf X']^{-1}\text{ exists}\right\}$$ - The model is \(\theta=(\boldsymbol\beta,F)\). - \(\boldsymbol\beta\in\mathbb R^k\); - \(F\) is the c.d.f of a distribution of \((\mathbf X,u)\). - \(G\) is such that \(Y=\mathbf X'\boldsymbol\beta+u\). - \(g\) is the distribution of \(\mathbf W=(Y,\mathbf X)\) observed from data. - The parameter of interest is \(\rho=\pi((\boldsymbol\beta,F))=\boldsymbol\beta\).

Denote the distribution of the observables derived from the model \(\theta\) by \(g_\theta\), i.e. \(g_\theta\equiv\mathbb P_F(\mathbf X'\boldsymbol\beta+u\le Y,\mathbf X\le\mathbf x)\) where \(\mathbb P_F(\mathbf X'\boldsymbol\beta+u\le Y,\mathbf X\le\mathbf x)\) is the distribution of \(\mathbf W=(Y,\mathbf X)\) implied by \(\theta=(\boldsymbol\beta,F)\).

In this linear regression example, suppose that \(\rho=\boldsymbol\beta\in\Pi^\star\). If for \(\forall g\in G\), we have \(\rho=\boldsymbol\beta=\phi(g)\) for some function \(\phi(\cdot)\), then we say that parameter \(\rho\) is point identified.

The fundamental goal of empirical economics is to learn from data. But behind this fundamental goal is a more fundamental question of what constitutes admissible evidence. Researchers sometimes say that the data speaks for itself. This is never true. There are always preconceptions embedded more or less precisely into "empirical evidence." A central question is how to revise these preconceptions in light of new evidence. Let's look at a particular method for reacting to surprises, which is called abduction.

Let's look at the following analogy to better understand deduction, induction and abduction (the "white beans" syllogism):

1. Deduction
2. (a) Rule: all the beans from a bag are white.
3. (b) Case: these beans are from that bag.
4. (c) Therefore, result: these beans are white.
5. Induction
6. (a) Case: these beans are from this bag.
7. (b) Result: these beans are white.
8. (c) Therefore, rule: all the beans from this bag are white.
9. Abduction
10. (a) Rule: all the beans from a bag are white.
11. (b) Result: these beans are white.
12. (c) Therefore, case: these beans are from that bag.

Friedman's methodology (abduction) is: 1. Create theory 2. Test theory 3. Revise theory 4. Iterate

Cowles's structural approach (Popperian's view) is: 1. First decide on a set of all admissible hypotheses (called "model"). 2. Then, choose from that subset of hypotheses to find the one that is consistent with the evidence (called "identification").

The major difference is that Friedman imposes as less restrictions as possible on the structure of the model, and then look at the data to figure out what the model should be that is consistent with the data, so Friedman's approach involves a lot of dynamic revising and retesting of the model, and also requires cross-validation. However, Cowles believes that there should be a set of admissible models, so the true model must come from that subset of admissible models, which is determined even before touching the data. The problem is, though, if all hypotheses in that subset are rejected by the evidence, then it is not clear what to do at that stage, and it's impossible to learn from surprises. And, as Friedman pointed out, such subdivision into two steps (decide admissible set and then identify) can be arbitrary and problematic at sometimes, which is just a special case of the general abductive approach.

Abduction versus identification

Abduction is a direct challenge to the identification. In particular, identification analysis takes its class of admissible models as determined before an empirical investigation begins. And the identification paradigm does not give a procedure for revising models.

Abduction versus Bayesian

Abduction and Bayesian are similar in the sense that both involve dynamic revising and updating of the model after observing new data. But, Bayesian has no way to cope with the totally unexpected surprises as priors rule out "totally surprising facts." Total surprise is only in the domain of abduction.

Abduction and cross-validation

To justify the abduction process, we need to do cross-validation, which is basically testing the model using data sets that are independent of the one used to generate and revise the model. For example, we can randomly split the data set into two parts, then use one part to generate and revise the model and use the other part to test the model.

19.5.1 Random sampling

Suppose that each of \(\mathbf X_i\)'s is a vector of random variables that are i.i.d. across \(i\)'s with a joint p.d.f \(f(\mathbf X_i)\). Simple random sampling attains each \(\mathbf x_i\) as a realization of \(\mathbf X_i\)'s, with the probability of obtaining the sample as $$\prod_{i=1}^{N}\mathbb P(\mathbf x_i)$$

The sampling rule, which is defined below, for the random sampling is \(\omega(\mathbf x^\star)=1\) for \(\forall\mathbf x^\star\), and thus \(\omega^\star(\mathbf x^\star)=1\), which implies that $$g(\mathbf x^\star)=f(\mathbf x^\star)$$

19.5.2 Sampling rule

Denote the original random variable by \(\mathbf X\) (lowercase \(\mathbf x\) denotes each value of \(\mathbf X\)) whose p.d.f is \(f(\cdot)\), and the sampled data by \(\mathbf X^\star\) (lowercase \(\mathbf x^\star\) denotes each value of \(\mathbf X^\star\)).

Note that the denominator of (19.1) just normalizes \(g(\mathbf x^\star)\) to make it integrate up to 1 so that it is a well-defined density function. Alternatively, we can define \(\omega^\star(\mathbf x^\star)\) by $$\omega^\star(\mathbf x^\star)\equiv\frac{\omega(\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}$$ so that $$g(\mathbf x^\star)=\omega^\star(\mathbf x^\star)f(\mathbf x^\star)$$

Note that it is possible that our sampling rule may put \(\omega^\star(\mathbf x^\star)=0\) on some data points \(\mathbf x^\star\), which gives us the truncated sample discussed below.

19.5.3 Truncated sample and truncated random variable

• Truncated sample: suppose \(f(\cdot)\) is the density of a random variable \(\mathbf X\). We call it a truncated sample if the sample is (in the univariate case) obtained only from \((a,b)\) for some \(a,b\) that can be potentially infinite.
• If the sample is obtained from \((-\infty,b)\), then the resulted sample is a right truncation.
• If the sample is obtained from \((a,\infty)\), then the resulted sample is a left truncation.
• Truncated random variable: assume that \(\mathbf X\), the latent variable, can only be observed when it is within \((L,R)\), denote the observed random variable by \(\mathbf X^\star\), which is the truncated random variable, and we know that $$\mathbf X=\mathbf X^\star\text{ for }\mathbf X\in(L,R)$$
• Suppose that we know the true untruncated population density \(f(\mathbf X)\), then the density of truncated random variable \(\mathbf X^\star\), denoted by \(g(\mathbf X^\star)\), is $$g(\mathbf x^\star)=\frac{f(\mathbf x^\star)}{\int_L^R f(\mathbf x^\star)\,d\mathbf x^\star}$$ or equivalently, $$g(\mathbf x^\star)=\frac{\omega(\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}f(\mathbf x^\star)\text{ with }\omega(\mathbf x^\star)=0\text{ if }\mathbf x^\star\notin(L,R)$$
• Suppose instead that we know the density \(g(\mathbf X^\star)\) of the truncated random variable \(\mathbf X^\star\) but don't know the true untruncated density \(f(\mathbf X)\), then it is impossible to recover \(f(\mathbf X)\) because we know neither the probability weight of the observed part (i.e. \(\int_L^R f(\mathbf x^\star)\,d\mathbf x^\star\)), nor the density structure outside the observed part.

19.5.4 Censored sample

• Censored sample: same as in truncation, we can only observe \(\mathbf X\) in the range \((L,R)\), which is denoted by \(\mathbf X^\star\). But unlike in truncation where we don't even know the number of outside observations, here in censored sample we know the number of outside observations but we just don't know the values of those outside points.
• Censored random variable: suppose the random variable \(\mathbf X^\star\) can only take values in the range \((L,R)\) like the truncated random variable, but the probability weight of in-range values in the whole population is known, then \(\mathbf X^\star\) is a censored random variable.
• Intuitively:
• Truncation: for example, if you are fishing on the sea using a net with hole diameter \(d\), then the sample of fish sizes is left truncated because there will never be a fish of length less than \(d\) appearing in your sample.
• Censoring: for example, if you are using a scale with minimum of \(\underline w\) to weigh fish, then the sample of fish weights is censored because you know the number of outside sample points (i.e. those too light fish) but you don't know their weights.

We have two types of censoring: - Type I censoring: we only observe a variable if it is in a range, but the number of values outside that range is known. - For example, in the light bulb burnout experiment with certain number of bulbs, type I censored sample is obtained when we specify a fixed length of observation period, where the number of burnout is a random variable but the experiment time is fixed.

• Type II censoring: stop sampling if fixed proportion of sample is obtained.
• For example, in the light bulb burnout experiment, type II censored sample is obtained when we specify a fixed proportion of burnout as a stopping rule, where the number of burnout is fixed but the experiment time is a random variable.
• In both types, we know the number of left-out observations but just don't know their value (e.g. in the light bulb example the value of interest is the burnout time of each bulb).

19.5.5 Stratified sample

Suppose that we have two random variables \(\mathbf X\) and \(\mathbf Y\), and we focus on the conditional density \(f(\mathbf y\mid\mathbf x)\). We are interested in the distribution of \(\mathbf Y\) given each value \(\mathbf x\) of \(\mathbf X\). So, the distribution of \(\mathbf X\) is not of direct interest.

In that case, if samples are selected solely on the \(\mathbf X\) variables, i.e. \(\omega(\mathbf y,\mathbf x)=\omega(\mathbf x)\) for \(\forall\mathbf y,\mathbf x\), then the sample selected by the sampling rule \(\omega(\mathbf y,\mathbf x)\) still gives us valid inference of the population conditional density \(f(\mathbf y\mid\mathbf x)\). Such samples are called stratified samples (stratified by \(\mathbf X\)).

The relative frequencies of each stratum \(\mathbf X=\mathbf x\) might be biased, but the frequency (estimated density) within each stratum is unbiased if \(\omega(\mathbf y,\mathbf x)=\omega(\mathbf x)\), i.e. \(\mathbf Y\mid\mathbf X\) is randomly sampled within each stratum \(\mathbf X=\mathbf x\). See below for the mathematical argument:

$$\begin{aligned}g(\mathbf y^\star,\mathbf x^\star)&=\frac{\omega(\mathbf y^\star,\mathbf x^\star)f(\mathbf y^\star,\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}\\&=\frac{\omega(\mathbf x^\star)f(\mathbf y^\star,\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}\\&=f(\mathbf y^\star\mid\mathbf x^\star)f(\mathbf x^\star)\frac{\omega(\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}\end{aligned}$$

and $$g(\mathbf x^\star)=\frac{\omega(\mathbf x^\star)f(\mathbf x^\star)}{\int\omega(\mathbf x^\star)f(\mathbf x^\star)\,d\mathbf x^\star}$$

So, $$g(\mathbf y^\star\mid\mathbf x^\star)=\frac{g(\mathbf y^\star,\mathbf x^\star)}{g(\mathbf x^\star)}=f(\mathbf y^\star\mid\mathbf x^\star)$$

i.e. the conditional density of the sampled data \(g(\mathbf y^\star\mid\mathbf x^\star)\) equals exactly the population conditional density \(f(\mathbf y^\star\mid\mathbf x^\star)\), so stratified sampling gives valid inference.

In this subsection, we hope to establish a clear distinction between ex-ante and ex-post inference. In particular, classical inference is ex-ante. In advance of estimation, we need to specify which tests we want to run. The likelihood principle and Bayesian inference are ex-post. That is, testing happens after we see the data.

19.6.1 Pure significance tests

These tests focus exclusively on a null hypothesis. In these tests, \(\mathbf Y=(Y_1,\dots,Y_N)\) are observations from a sample and \(t(\mathbf Y)\) is a test statistic.

If 1. we know the distribution of \(t(\mathbf Y)\) under null hypothesis \(H_0\); 2. the larger the value of \(t(\mathbf Y)\), the more the evidence against null hypothesis \(H_0\);

then a low value of \(\mathbb P_{\text{observed}}\) is strong evidence against the null hypothesis, where $$\mathbb P_{\text{observed}}=\mathbb P\!\left(t\ge t_{\text{observed}}\mid H_0\text{ is true}\right)$$

\(\mathbb P_{\text{observed}}\) is the \(p\)-value, which is the probability that \(t_{\text{observed}}\) would occur given that \(H_0\) is true. So, a low \(p\)-value implies that \(t_{\text{observed}}\) should not happen if \(H_0\) is correct, which means that \(H_0\) is very likely to be false given that we have already observed \(t_{\text{observed}}\).

19.6.2 Sample size and rejection probability

There's a mechanical relationship between increasing sample size and the rejection of a hypothesis. In particular, if the null is not exactly true, then we get the probability of rejections for a fixed significance level approaching 1 very fast as sample size increases.

For example, suppose we run the test: $$H_0:\bar X\sim N\!\left(\mu_0,\frac{\sigma^2}{T}\right),\qquad H_A:\bar X\sim N\!\left(\mu_A,\frac{\sigma^2}{T}\right)$$ where we assume that \(\sigma^2\) is known. For any critical value \(c\) for rejection, we can find its corresponding significance level \(\alpha(c)\) by $$\mathbb P\!\left(\frac{\bar X-\mu_0}{\sqrt{\sigma^2/T}}>\frac{c-\mu_0}{\sqrt{\sigma^2/T}}\right)=\alpha(c)$$

Reversely, for any significance level \(\alpha\), we can find its corresponding critical value \(c(\alpha)\) by $$c(\alpha)=\mu_0+\sqrt{\frac{\sigma^2}{T}}\,\Phi^{-1}(1-\alpha)$$ where \(\Phi(\cdot)\) is the c.d.f. of standard normal distribution.

Now, consider the case where we fix the rejection rule, i.e. a fixed critical value \(c\). For a sample size \(T\), suppose the true mean is \(\mu_{\text{true}}\), then the probability of rejection is

$$\begin{aligned}\mathbb P(\bar X>c)&=\mathbb P\!\left(\frac{\bar X-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}>\frac{c-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}\right)\\&=\mathbb P\!\left(\frac{\bar X-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}>\frac{\mu_0-\mu_{\text{true}}+\sqrt{\sigma^2/T}\,\Phi^{-1}(1-\alpha)}{\sqrt{\sigma^2/T}}\right)\\&=\mathbb P\!\left(\frac{\bar X-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}>\frac{\mu_0-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}+\Phi^{-1}(1-\alpha)\right)\\&=\mathbb P\!\left(\frac{\bar X-\mu_{\text{true}}}{\sqrt{\sigma^2/T}}>\frac{\sqrt T(\mu_0-\mu_{\text{true}})}{\sigma}+\Phi^{-1}(1-\alpha)\right)\\&=\alpha\quad\text{when }\mu_0=\mu_A\end{aligned}$$

However, if \(\mu_{\text{true}}>\mu_0\), then the probability \(\mathbb P(\bar X>c)\) of rejection goes to one (since \(\frac{\sqrt T(\mu_0-\mu_{\text{true}})}{\sigma}\to-\infty\) as \(T\to\infty\)).

19.6.3 Classical hypothesis testing

Classical hypothesis testing is the frequentist's view. Basically, it assumes that the test can be repeated many times, and the fraction of rejection when the null is true should asymptotically approaches the probability of making type I error, i.e. significance error. So, classical hypothesis testing is based on a long run justification.

However, some criticism is that although such testing approach can be consistent in level (i.e. probability of type I error approaches significance level), consistency is not always a meaningful concept because such testing approach is likely to have poor finite sample properties.

19.6.4 Likelihood principle

The likelihood principle believes that - all the information is in the sample; - we should look at the likelihood as the best summary of the sample.

The basic idea is that we can only observe one data set. So, when comparing two candidate hypotheses, we should accept the hypothesis that generates higher likelihood for the observed data set (i.e. the probability of obtaining the observed data set). The intuition is that given the observation has already taken place, we should believe in the hypothesis that gives the highest probability of the realized observation.

19.6.5 Bayesian principle

The Bayesian principle - uses prior information (hypothesis or model specification) in conjunction with sample information; - the imposed prior information can be subjective or objective.

In conclusion: - classical hypothesis testing uses no data in the hypothesis generating process; - likelihood approach uses only data in the likelihood-maximizing hypothesis generating process; - Bayesian approach uses the combination of prior information and the data information in the hypothesis generating process.