One item, four ways it can land. “Was the outcome assessor blinded?”The mark is the answer the people who want the data would give.
valid (unbiased)
not valid (biased)
reliable (low variance)
Yes, every time. And you would say yes too. Trust it.
Yes, every time. The codebook never said whom. Every reader agrees. Every reader is wrong.
not reliable (high variance)
Yes, no, yes. The readers split around your answer. Something in the item is unclear.
Yes, no, no. Split, and off the mark. Unclear and misread.
to find unreliable itemsRun the same codebook through diverse models. Where they disagree, the cell is flagged. The fix is a rule in the codebook, not a hand on the cell.
to find invalid itemsRead the codebook and ask whether it says what you mean. Then label a handful of documents yourself and see whether the readers land where you did.
Where readers disagree, look. Where they agree, ask whether you would.
@karlrohe
The mark is what the people who want the data would say. Not what a model would say. Not what an expert would say. What you would say if you read the document yourself. Reliable: ask the item again and get the same answer. Valid: the answer is the one you meant. A miss is never the model alone. It is the model plus the codebook, read against your intent. The blinding item is the usual shape: the instruction was true as written and silent about the thing that mattered.
The two checks are not symmetric. The flag you can buy: run the same codebook through readers from different model families, and an unreliable item shows up as a split. What you do with the flag is craft. Remove the ambiguity with a rule written into the codebook, one that applies to every document, and leave the cells alone. Or leave the flag standing. Some splits have no policy behind them, and a rule written for one paper makes the codebook worse, not sharper. Validity you cannot buy. The mark lives with you. Read the codebook and decide whether you agree with it. Then label a handful of documents yourself and see whether the readers land where you did. Where they do, you have earned some trust. Where they don’t, you have found the item to fix.