Automation is now cheap. Scientific-grade data isn’t.
AI now provides automation. It does not necessarily provide scientific-grade data; that requires you.
Systematization
One written rule per question, applied to every document.
toward engineering
Readers run. Cells fill. The same machinery for every project.
This is what AI just made cheap.
toward measurement
Reliable: ask again, get the same answer. Valid: the answer is the one you meant.
This is what a systematic review does by hand, with no automation at all.
AI moved one branch a long way. It did not move the other.
@karlrohe
Systematization is upstream of both automation and scientific-grade data. Systematization is one written rule per question, applied to every document. You cannot automate an extraction that you have not specified. Similarly, there is no scientific question to ask if you do not have a system to evaluate. AI gives you the automation cheaply. The rule you still have to write.
Systematization is necessary on both sides and sufficient on neither. Scientific-grade data is reliable and valid, and you can show both. A process can follow its rules exactly and still be unreliable, because a rule can be read two ways. It can follow its rules exactly and still be invalid, because the rule asks the wrong thing. Systematization enables reliability and validity. It does not deliver them. It does make the process transparent, so anyone can read the rules, and replicable, so anyone can run them.
A systematic review is measurement without automation. Human coding teams have always worked this way: a coding manual, a training session, an adjudication meeting. No automation at all. It is slow. In one randomized comparison, extracting and verifying a single study took 107 minutes, and 172 with dual independent extraction.1 That is what measurement costs when you pay for it by hand.
The trap is to take the automation and skip the system. The codebook is where the system lives. Everything a trained coder carries in their head has to be on the page, because the AI reader has nothing else. That is why the rest of this series is about the codebook.
sources1Li et al. 2019, Journal of Clinical Epidemiology 115:77–89.