“The model version changed and my pipeline broke.”
“I don't trust it for anything that matters.”
Clearly, we have a long ways to go.
@karlrohe
AI gets used for this work in a lot of different ways, and none of them is very developed yet. That is the real problem behind the quotes on this card. Every objection is something an undeveloped practice produces. None of them is evidence that the work cannot be done.
A mint is an attempt at a developed way of working. The question is written down before it is asked. It is asked of every document the same way. Several independent readers answer it. The evidence stays attached to the answer. Where the readers split, you are told.
Trust comes from the arrangement, not from the model. No one technique on that list does much alone. Together they change what you are able to say about the result. You are no longer reporting an impression of how the AI did. You are reporting a measurement with a written definition and a known split rate.
Trust you cannot hand to someone else is not worth much in research. You can give a reviewer the codebook and the cells and let them check the read themselves. Whatever confidence you built travels with the artifact instead of staying with you.
A good mint does a lot behind the scenes, and it gives you a clear way to look at what it did. The first half earns the trust. The second half is what lets you pass it on. Each objection on this card has an answer in that arrangement. The next cards are those answers.