AI fills the spreadsheet, and every cell carries its evidence.
Every row is a document. Every column is a question. Every cell is an answer linked to its evidence.
There are two reasons that each cell carries its chain of thought:
For the AI reader. The evidence is a quote from the document and the reasoning that turned that quote into an answer. Writing the chain prevents hallucination — context the reader can't skip.
For you. Click any cell and the full chain is there to audit: the quote, the reasoning, the assumptions, the answer.
One cell, unfolded.
@karlrohe
Every column holds one item. A codebook is a list of items, and it is the same list for every document in your corpus.
definitions
item — one codebook question as it lands on one document. Not the question in the abstract. This question, this document, one answer.
reader — the model that reads one document and answers one item. Data Mint can send the same document to several readers. Any model can be a reader.
A cell holds more than an answer. Click it and it unfolds. The full statement of the question, so you can see what was actually asked. The full answer, when it does not fit in the cell. The key quotes the reader worked from. The reasoning that turned those quotes into an answer. The assumptions it had to make along the way.
You can ask a reader how certain it is, and what you get back is a vibe. It is not bad. It is not great. Don't take it to the bank.
Every piece of that metadata is work the reader does before it answers. It has to find a quote. It has to say something about that quote. It has to reason from the quote to the answer, and name what it assumed. Answering a codebook item is usually several steps. It is not read the document, then answer.
Breaking the work into steps is what removes hallucination. In my experience I have not really seen hallucinations. I see ambiguities, lots of ambiguities. Making things up is not the failure I run into.
The failure I do run into is a cell that comes back marked failed. Nothing was answered. You run it again.
The metadata is also how you find out whether the reader understood the question. The answer on its own tells you very little. What you want to know is how it got there. Give a new RA in your lab a task, and you do not just check whether they got the right answer. Checking is harder than it sounds. Often you do not know the right answer either. You don't have the answer key yet.
So instead you ask them to talk about what they did. From how they talk about it you can tell whether they understood the task, and whether they did it well. And you build a relationship with the person in the process. Opening a cell and reading its quotes and reasoning is the same move.
An answer can be right without showing that the question was understood. The process of arriving at it does show that. That is the feeling I am after when you open a cell. I mean understood in the ordinary, colloquial sense, the way you would say it of a colleague.
