highlight · part 0 · 2

Making data is a scientific endeavor.

How do we know whether the data extracted is valid and reliable? This has nothing to do with AI. It has been the hard problem for a century.

1850
Textual criticism. Karl Lachmann's stemmatic method for reconstructing a text from many flawed documentary witnesses — compare, track shared errors, reconstruct the most defensible reading. The deep ancestor of extraction with an audit trail.
1927
–1952
Content analysis. Harold Lasswell makes political text into analyzable evidence. Bernard Berelson's Content Analysis in Communication Research (1952) formalizes the codebook — coding texts into variables by explicit rules.
1955
–1960
The math of agreement. William A. Scott's π (1955), then Jacob Cohen's κ (1960): the first chance-corrected measures of whether two readers extracted the same thing — beyond luck.
1993
The protocol. Iain Chalmers founds the Cochrane Collaboration. Duplicate extraction, standardised critical appraisal, audit trails — extraction from documents becomes a regulated craft.
2022+
A new reader. Large language models make the codebook executable at scale. Everything upstream — construct definition, codebook design, adjudication, agreement, provenance — unchanged.

the open question

How can we tell whether the AI reader is better or worse than a human reader, and what does that even mean? Humans disagree with each other. Validating a reader without a gold standard is part of my research program, and Data Mint is where it is tested.

The reader is new. The craft of checking the read is older than the Turing test.

@karlrohe

Reliability and validity are different questions, and both are older than the machine. Reliability: two readers, the same document, the same codebook. Do they give the same answer? Validity: is the answer the thing you meant to measure? Statisticians call the first low variance and the second low bias. The timeline carries both. Lachmann reconstructs the text that was actually written. Cochrane asks whether a study measured what it claims. Scott and Cohen ask only whether two readers agree.

Cohen’s κ is one of the better-known reliability numbers. It adjusts raw agreement for chance, using the marginal rate of each answer.1 κ says nothing about validity. Two readers can agree on every abstract and both be “wrong,” because they read the same unclear instruction the same way. No agreement statistic between readers can certify the instruction the readers were given.

How can we tell whether an AI reader is better or worse than a human one, and what would that even mean? Humans disagree with each other. The gold standard is usually a third human. Validating a reader without a gold standard, because the humans disagree too, is the research program behind Data Mint. This card does not answer the question. The next cards say what can be checked, and how.

sources1Cohen 1960, Educational and Psychological Measurement 20.

see alsowhat disagreement can’t tell you— Agreement is not the same as being right.codebook as estimator— Reliability and validity, named as variance and bias.validation is layered— What we actually check, layer by layer.

Codebooks were the answer before we knew the question was AI. datamint.ing/learn/highlights/has-nothing-to-do-with-ai