Disagreement is a gradient.
Revision is a step.
Codebook refinement is coordinate descent — in English.
Each round, the readers point at which edit matters most — not by proposing one, but by diverging exactly where the codebook is silent. Honest note: the "gradient" is a finite-difference diagnostic, not a derivative. What you're actually doing is coordinate descent — one field at a time — with a held-out paper set to keep you from overfitting the pilot.
loss
total disagreement across readers (field-weighted)
step direction
which edit helps most next
step size
edit magnitude per round
validation set
held-out papers — never refined against
early stopping
no new disagreement types this round
local minimum
shallow consensus — agreement on an easier question
cf.Settles (2009) Active Learning · Lakshminarayanan, Pritzel & Blundell (2017) · Saldaña (2021) · Glaser & Strauss (1967) · Breiman (2001) Two Cultures
Each round can move the codebook downhill.
@karlrohe