Validation and Calibration: The Hard Part
Agent-based models are criticised for being unfalsifiable. The criticism is not unfair, but it describes bad practice rather than the method. The discipline exists; it is just more work than fitting a regression.
The core problem is identification. A model with heterogeneous adaptive agents has many degrees of freedom, and several genuinely different mechanisms can reproduce the same aggregate pattern. Reproducing a stylised fact therefore provides much weaker support for a mechanism than it feels like it does.
Fitting is not validating
Fitting means finding parameters that reproduce data the model was tuned on. Validating means the model reproduces something it was not tuned on. The distinction is standard everywhere else in empirical work and is skipped surprisingly often in ABM, usually because the available data was all used for calibration.
The practical version: hold something back. A time period, a region, a subgroup, or a moment of the distribution that played no part in calibration. If nothing can be held back, say so explicitly rather than presenting a fit as a test.
Levels of validation, weakest to strongest
- Face validity. The rules are behaviourally plausible to domain experts. Necessary, and close to worthless on its own.
- Stylised-fact reproduction. The model produces the qualitative regularities of the system - fat-tailed returns, skewed firm sizes, clustered adoption. Weak evidence, because many mechanisms produce these.
- Cross-validation. Calibrate on one subset, test on another. This is where meaningful evidence starts.
- Out-of-sample prediction. The model predicts a period or case it has never seen. Rare, and the strongest ordinary evidence available.
- Mechanism validation. The intermediate quantities the model produces - not just the headline output - match independent measurements. The most demanding test, and the one that distinguishes a mechanism from a curve fit.
Calibration methods
Direct estimation of behavioural parameters from micro data is the cleanest route when the data exists, because it removes those parameters from the fitting problem entirely. Where it does not, simulated method of moments is the workhorse: choose target moments, search the parameter space for the setting that minimises the distance between simulated and empirical moments. Bayesian approaches extend this by returning a posterior distribution over parameters rather than a point, which is more honest about the identification problem, at considerable computational cost.
Whichever is used, report the whole picture: how many parameters were free, how many moments were targeted, and how sharply the objective function identified the optimum. A flat objective surface means the data does not pin the parameter down, and that is a result worth reporting rather than hiding behind a point estimate.
Sensitivity analysis is not optional
One-at-a-time sensitivity analysis is standard and misleading, because it cannot detect interactions between parameters, which is where the interesting behaviour of a complex model lives. Global methods - variance-based decomposition, or a well-designed sampling scheme over the joint space - cost more and tell you which parameters actually drive the output. If a conclusion holds only in a narrow region of parameter space, that is the conclusion.
Reporting standards
Structured protocols for describing models exist precisely because free-form descriptions turned out to be irreproducible. Whichever protocol is used, the requirement is the same: someone else must be able to rebuild the model from the description alone. In practice this means publishing the code, the parameter settings, the random seeds and the analysis scripts. A model that cannot be rerun is a claim, not a result.
What honest limitations look like
Every model should state which questions it is not fit to answer. A model calibrated to reproduce aggregate dynamics is usually not fit to evaluate a policy targeted at a subgroup whose behaviour was never separately identified. Saying so costs nothing and is the difference between a research instrument and a persuasion device. The same standard applies to the empirical claims discussed in big data and public policy, and to the structural models described in financial instability and crises.