highlight · part 0 · 7

Every codebook question will meet a document it wasn't written for.

Every codebook question will meet a document it wasn’t written for.

Ask a clear question of a document and you get an answer. Ask an unclear one and you get a guess.

A cell comes out right when three things work together:

The document says it plainly.
The codebook asks it clearly.
The reader attends to both.

When any one of the three falls short, careful readers can defend different answers. That is ambiguity, and it is the fundamental problem of this work.

Lots of projects fail here. The ambiguities were there. Nobody saw them.

The good news: readers who disagree show you where the ambiguity is. Data Mint runs several readers, so the disagreement is visible.

Resolving them is craft. The reward is real: you understand your question better than you did before. Your measurement gets sharper. Every downstream answer gets stronger. What follows is mostly about this craft.

@karlrohe

Two conjectures carry the rest of this series. Agreement licenses trust: where readers agree, accept the extraction without human verification. Disagreement directs attention: where readers disagree, the cell needs a human decision, and the codebook may need a clarifying note. Neither conjecture is a fact about AI. Both are design goals. Data Mint is designed so that agreement can license trust and disagreement directs attention. That design is what puts verification effort where the ambiguity is.

The loop earns the first conjecture, and a decision ends it. Readers agree where the codebook has been through the loop in Part III. Each round removes one kind of disagreement. The kinds that remain are cells a person will read once the whole corpus runs. Five pilot documents cannot tell you how many such cells there will be. They can tell you what kind they are. You stop refining when the kinds still splitting have no policy behind them. A rule for one paper's quirk does not sharpen the codebook; it makes it a list of exceptions. What remains is a residual. Whether the team reads it or accepts it as error is a second decision, and it is theirs.

Agreement buys reliability, not validity. Three readers can agree on the same wrong reading of a question. The reading that counts is the one you meant, and only you have it. That gap does not close on its own. It is a separate claim with its own check. The next card is where the check starts.