Think of a model as a set of assumptions (do I owe Quine and Duhem a citation here?).

These assumptions produce a set of predictions. The “predictions” need not be about the future. They may be explanations of events or patterns of events observed in the past.

When a model’s predictions turn out to be true, the model should be rated highly if and only if it is very difficult to obtain those predictions from other models. In order to have the best chance of being rated highly, a model should make many predictions, at least some of which are contrary to predictions made by other models.

When a model’s predictions turn out to be false, the model should be rated highly if and only if the failure is interesting. What does that mean?

Suppose that we classify the assumptions in a model as either verified or doubtful. A verified assumption is an assumption that can be demonstrated to hold true in the situation in which a prediction is being made. A doubtful assumption is one that either cannot be verified or is even known to be false.

If a model makes false predictions, then one or more of the doubtful assumptions does not hold. If there are many doubtful assumptions, then we do not know which one, and we learn almost nothing from the prediction failure. At the other extreme, if there is only one doubtful assumption, then a prediction failure is interesting in the sense that it tells us that this doubtful assumption does not hold and this has important consequences.

So you want to try to eliminate doubtful assumptions. One way to do this is to generalize a model. That is, if your model assumes that there are two factors of production, can you get the same result if there are many factors of production? If so, then you have eliminated the assumption of two factors of production as a doubtful assumption. Another way to eliminate doubtful assumptions is to demonstrate using data that the assumptions hold.

In short, we should rate highly a model when

(a) if almost all of its main predictions are true, at least some of them differ from the predictions made by other models

(b) if some important main predictions are false, there is a very limited set of doubtful assumptions that might explain the false predictions

My guess is that very few economic models could be rated highly by these criteria.

The term “model” can mean many things.

1. To an engineer, a model might be something like a flight simulator. It attempts to replicate actual conditions so well that it can be used to train a pilot who then moves on to operate in the real world. Economists sometimes treat their models this way. Think of models used to forecast the impact of tax changes, or think of Jonathan Gruber’s model used to predict the impact of Obamacare on the health insurance market. In my view, most of the time the accuracy of these sorts of exercises is often far over-stated.

2. In economics, models are often used for a different purpose. The economist writes down a model in order to demonstrate or clarify the connection between assumptions and conclusions. The typical result is a conclusion that states

All other things equal, if the assumptions of this model hold, then we will observe that when X happens, Y happens.

For example, X could stand for “a firm raises its price” and Y could stand for “the demand for its product goes down.”

3. Model failure is usually more interesting than model success.

Suppose that we observe a situation where X happens and Y happens. Does that confirm the model? Because there typically are other models that can explain such a pattern, we usually do not draw strong conclusions based on such evidence.

However, suppose that we observe a situation where X happens and Y does not happen. Does that refute the model? I would say that what it refutes is the prefatory clause “other things equal, if the assumptions of this model hold.” That is, we may conclude that other things were not equal or that the assumptions of the model do not hold.

In my book, I use the example of a college that raised its tuition and experienced a subsequent increase in applications for admission. I do not say that the law of demand fails to hold. Instead, I say that other things were not equal. In particular, the college also raised its level of financial aid. Thus, although its “list price” went up, the discounted price faced by many applicants went down.

The prefatory clause in economic models makes it difficult to draw scientific conclusions from real-world observations. When X and Y occur as predicted, we cannot confirm any one model, because other models are consistent with result. When they do not occur as predicted, we only know that the prefatory clause was violated–the assumptions of the model were not met or other things were not equal.

Often, we cannot say anything very interesting about which assumptions were not met or which things were not equal. For example, the model of an aggregate production function is used to predict that differences in output per worker will be proportionate to differences in capital per worker. When this fails, there are many possible reasons: workers may differ in their human capital; physical capital may not be measured or aggregated correctly; output may not be measured or aggregated correctly; institutional differences may matter. etc.

In fact, the primary use of the aggregate production function model is to examine its failure, which is called “the residual.” Economists place an interpretation on this residual, calling it “total factor productivity.” They interpret the rate of change in this residual over time as “productivity growth.” They interpret the change in the rate of change in this residual as “change in the trend rate of productivity growth.”