You want to extract data from many documents — screen abstracts, code findings, assess bias. You want AI to help. But almost every task is judgment. Even simple extractions often hide expert judgment, and AI is bad at judgment.
Your task
To make systematic judgments, you need to decompose them into a sequence of reading comprehension tasks. Each reading task is one key step in the judgment. It takes skill. A codebook is where you write it down — and it is the prompt you give to the reader.
The good news
This is not new. It is not about AI. Complex judgments become more reliable when they are decomposed into simpler, observable parts.
JUDGING
AI collapses
READING
AI thrives
Is the sample size adequate?
Did the authors describe a power calculation for their planned sample size?
Is the coding reliable?
Did the authors report an inter-rater agreement statistic?
Is the effect clinically meaningful?
Did the authors define a minimum clinically important difference?
a pattern across disciplines, long before AI
Six decompositions, in five fields, the first in 1954.
Meehl (1954)→clinical diagnosis, decomposed into weighted rules on observable traits.
Bueno de Mesquita (1981)→conflict outcomes, decomposed into four estimated variables.
Altman (1994)→medical research quality, reframed as checkable technical obligations.
Gawande (2009)→surgical expertise, decomposed into pre-op checklist items.
Sterne et al. (2019)→RoB 2: risk of bias, decomposed into five domains of signaling questions.
Kahneman et al. (2021)→noisy judgment, decomposed via decision hygiene.
The codebook is the bridge from reading to judgment.
cf.Descartes (1637) · Polya (1945) · Meehl (1954) · Simon (1962) · Alexander (1977) · Bueno de Mesquita (1981) · Altman (1994) · Richardson (1995) · Begg et al. (1996) · Kahneman & Frederick (2002) · Gawande (2009) · Tetlock (2015) · Sterne et al. (2019) · Kahneman et al. (2021)
@karlrohe
I have been working with these models since about 2023, and the split I saw then still holds. I started with GPT-3.5, and a little before that with GPT-3. There was a seed of truth in the split back then. It is kind of still true now.
By reading I mean one narrow thing: read a document and answer a question. That is the whole process. Given clear instructions, the models are very good at it.
definitions
reading question — a question a careful person could answer from the document in front of them. Did the authors report a power calculation? Did they name a minimum clinically important difference?
judgment question — a question whose answer depends on things that are not in the document. Is the sample size adequate? Is the coding reliable?
"Is the sample size adequate?" cannot be answered from the paper. Ask a professional statistician and you get follow-up questions instead of an answer. What effect size do you expect? What did your earlier experiments show? What is your experience with those experiments? They would have a million questions.
Models are getting better at asking those follow-up questions, and that is a real change since 2023. Early on you got the coffee-room answer. A hundred, sure. A thousand, that sounds like a lot. Ten thousand, great, that is a really good sample size. Colloquial, and not thinking about it rigorously.
"Is the coding reliable?" is the same kind of question. I do not expect AI to be very good at it. It can help. It is still a judgment question.
The reader needs to know what you were thinking when you wrote the item. Some of what it needs is public and simply not in front of the reader. Some of it only you have. Either way the reader is short of it, and the question comes off as ambiguity. It just does not know.
Meehl's case was that a written rule applied to observable traits beats an expert holding the same evidence in their head. Forty-six years of studies later, a meta-analysis found the same direction: mechanical prediction ran about ten percent more accurate than clinical prediction, across studies of human health and behavior.1
source1Grove et al. 2000, Psychological Assessment 12(1), 19–30: "on average, mechanical-prediction techniques were about 10% more accurate than clinical predictions."