The codebook is the method. Models are the execution layer.
If reviewers can't read it, it's not a method.
fine-tuned model
model-v2.ckpt4.7 GB
training dataprivate
hyperparamslost
decision ruleopaque
English codebook
##intervention_typeQuestion: What was the main intervention?
Description: The therapy, medication …
Example answer: "CBT, 12 weekly 60-min sessions, group format."
Default if missing: "Not specified"
✗can't appear in methods
✓fits in methods
✗deprecates on next upgrade
✓survives model updates
✗opaque to peer review
✓readable by reviewers
✗operational definition gone
✓decision rule preserved
A team fine-tunes a classifier to extract whether RCTs report allocation concealment. Two years later the model endpoint deprecates, the training set can't be shared for licensing, and a reviewer asks why "central randomization" counted in some trials but not others.
Performance numbers survive. The operational definition is gone.
cf.Krippendorff (2004) · Mitchell et al. (2019) Model Cards · PRISMA 2020 · Peng (2011) · Lundberg & Stewart (2021)
The codebook defines the decision rule. Models are the execution layer.
@karlrohe
I am interested in which model you used, and I think that interest is right. The open-source models, the tiny ones, the massive ones, the expensive ones, the closed ones, the Chinese ones. That zoo is genuinely interesting, and people want to know.
Some visual tasks are still hard, for open and closed models alike.
For text documents, vision aside, nearly any current model above the very small ones can answer a well-specified codebook item. Open source or closed, either can answer it. There are differences here and there. The specification is doing the work.
The codebook is about ten times more important than the model. Ambiguities in the codebook manifest so much more quickly than a model failing to understand something.
That claim has a date on it. A year ago the open models were good enough. The closed models were roughly a year ahead of that. Not uniformly, and not all of them. Two years ago this was less true.
Model choice is my job. I definitely have my preferences among models. I test them, quickly, in ways that are quite different from how other people test them, and the ones I love go into Data Mint. The model matters. It is not where your attention should go.
Reasoning capability is the wrong axis to shop on. If an item needs reasoning beyond a simple step, decompose it into several items in the codebook. A single hard step is not that reliable even for people. Complex judgments become more reliable when they are decomposed into simpler, observable parts.
You do not want that decomposition happening behind the scenes in a black box. It needs to be your job. That is how you get scientific-grade data.
Your codebook is far more interesting to me than your model. As a researcher I want to see how you specified things, not that you used GPT-5.6 Sol Ultramax. I don't care. I want to know the questions you asked.