Highlights — Data Mint

Part 0 — The setup

A huge and growing task. An old hard problem. A new possibility. Trust not yet earned. No shortcut to the specification. The fundamental problem at its heart. And the trick that surfaces it.

Turning documents into data.

Read documents. Produce structured data. The task is everywhere — and the demand is growing.

How do we know if the data extracted is valid and reliable?

This has nothing to do with AI. The craft of validating extraction from documents is old.

What if rigor was easy to scale?

If you can do it once with AI, you can do it as many times as you want. The hard part is the codebook.

Trust hasn't been deserved.

Documented failure modes from the validation literature. Hallucinations, drift, lost-in-the-middle, and more.

Garbage in, garbage out.

Engineers got let off the hook by smarter models. Scientists can't outsource the specification.

The fundamental problem we face.

Most cells fill cleanly. Some resist. Ambiguity is the work.

Data Mint identifies ambiguities with model diversity.

Diverse readers turn hidden bias into visible disagreement.

Part I — Why codebooks + AI is the combination

The artifact. The reader. The chain you publish. The codebook that travels.

A spreadsheet, filled in by AI.

Every row is a document. Every column is a question. Every cell is an answer linked to its evidence.

AI: bad at judgment, great at reading.

Decomposition is an old craft. AI is a new reader.

The whole chain, in three parts.

What you wrote. What you read. What you want.

Publish the codebook. Not the weights.

If reviewers can't read it, it's not a method.

Your codebook becomes someone else's starting point.

Share the URL. Another lab can run it, refine it, or extend it.

Part II — The codebook: anatomy and craft

What a codebook is. The documents it reads. The craft of writing a single field well.

What a codebook is.

A named list of questions. Each with its own instructions.

Not all documents read the same.

All four can become data with a codebook.

Good instructions have good examples. So do good codebooks.

Definitions define the space. Examples populate it.

Silence isn't No.

"Not reported" is its own answer — and usually the most common one.

One cell. Many shapes.

A number. A code. A paragraph. A list. A whole linked table.

Part III — Refining codebooks

Disagreement is the signal. AI readers surface it cheaply. The codebook responds.

Not every document fits the grid.

No hallucination — real quotes, valid reasoning. Still they don't agree. Three craft moves: sharpen, scaffold, relax.

Disagreements should inspire refinements to the codebook.

The ambiguity isn't in the paper; it's in what the codebook asked. And the codebook is what decides.

The codebook is never right on the first cut.

Four moves, repeated until the cracks close.

Scaffolding. Wordsmithing. Examples.

Three fixes for a failing field. The right one depends on why it failed.

Part IV — Theory

What a question is, as a measurement. Refinement as coordinate descent. The limits of what disagreement can tell you.

Every question has a bias and variance.

Each question is a measurement. Two ways a measurement can fail.

Disagreement is a gradient. Revision is a step.

Codebook refinement is coordinate descent — in English.

What disagreement cannot tell you.

High agreement doesn't mean you measured the right thing.

Part V — A worked example

One codebook. Two batches. Three drafts. The question itself moved.

One example of what codebook refinement can look like.

Its scaffolding changed across three drafts. So did the papers we tested it on.

The first batch came easy.

Recently-published abstracts. Readers still argued.

The next batch was chosen to be hard.

Searched PubMed for confusing types of studies.

Some edge cases needed new fields.

Not every failure fits into the fields you already have.

The question moved too.

We started asking whether a paper was a randomized trial. We ended asking something narrower.

The codebook clarified the question.

A BMJ CONSORT screener, in three drafts.

Part VI — Trust, built in layers

The coda. Trust is what everything above has earned. Stance is what you bring to it.

You can't score a relationship.

Trust in your data comes from carefully crafting the codebook. This relationship cannot be reduced to a number.

Validation is trust, built in layers.

No single layer is sufficient. Seven we build on, at Data Mint.

Data is neutral. I am not.

Every number got there by a choice. We make those choices.

My writing is what I control.

The pipeline is commodity. The codebook is yours.