highlight · part 0 · 5

Your codebook plus your documents make your data.

What you wrote. What you read. What you want.


Your codebook

the questions and decision rules you wrote down

Your documents

your corpus — papers, transcripts, scans, records

Your data

each row is a document, each column is a question, each cell is an answer linked to evidence

You bring the codebook and the documents. Data Mint makes the data.

@karlrohe

The chain has three parts, and the first two are yours. You have a codebook. You have some documents. For every document you fill out the codebook answers and put them in a spreadsheet.

I claim it works for any collection of documents and any codebook. I am sure we can come up with counterexamples where it will not work. I am interested in those.

Reading here is not only text. I say reading again and again, and the models are multimodal. Extract data from clinical-trial papers and the numbers you want are in tables and figures. Those are visual. Data Mint has Visual for that, and there are vision models that will read them.

A document can be a page of cursive in a bound volume. I am working with a project that has scans of historical ledgers. People writing by hand, in huge books, registering women to vote in Boston in the 1920s. It is beautiful script.

A page spread from a 1920s City of Boston register of women voters: printed column headings over ruled lines filled in with cursive handwriting.
A 1920s Boston voter-registration ledger. Cursive, in a bound volume. A crazy example of a codebook and documents.1

1City of Boston, Register of Women Voters, 1921 (ward 10, vol. 1A). Page-level view; names not legible at this size.

New problems arise in these cases, and those are the ones I am most interested in. Data Mint handles that class of document under the name Squint. The abstraction does not change. You need a codebook, and you have documents.