highlight · part 0 · 6

The specification is the science.

Engineers moved on from prompt engineering. Scientists cannot.

When ChatGPT first came out, engineers obsessed about prompt engineering. Then models got smart enough to guess what they meant. The discipline faded.

Scientists can't take that shortcut. We can't ask AI to guess what we meant by "the intervention." We can't trust it to fill in our intent. Measurement is the work of being precise about what we measure.

The codebook is the specification.

Bad specification, bad data. No model is smart enough to fix that.

And no model should be. Whatever you leave ambiguous, the model decides for you.

@karlrohe

Prompt engineering faded because chatbots learned to guess what you meant. A chatbot should guess. A measurement instrument should not. When a model extracts data, the written instructions are the instrument: the extraction form, the screening criteria, the coding manual. Content analysis has had a word for this for decades. The codebook: a list of questions, each with its definitions and its examples.

Readers guess differently. An instruction that can be read two ways will be read two ways. Both readings enter the data. Deciding what “the intervention” means for this study is the science.

see alsowhat is a codebook— The specification has a form, and this is it.good codebooks have examples— Examples pin down what the definitions leave open.the codebook makes the columns— The specification is what turns a pile of documents into a table.

Garbage in, garbage out. datamint.ing/learn/highlights/garbage-in-garbage-out