Correcting Toward the Wrong Target: LLM-Annotated Regressors Under Noisy Human Benchmarks
Publication Date
9-15-2025
Abstract
Two human coders scoring the same television advertisement rarely fully agree on whether it is a ``narrative'' ad, yet applied researchers who use large language models (LLMs) to annotate data typically validate the resulting annotations with a small human-labeled subset. The problem is that LLM annotation validation compares two noisy measurement systems, not just a labeler and truth. When the human labels are themselves noisy proxies for the underlying construct, correction methods that target the human-label coefficient pull LLM-based estimates toward an attenuated benchmark rather than toward the underlying-construct coefficient; the two differ by a factor equal to the human rating's reliability, and extra validation data does not close the gap. We confirm this result using a Monte Carlo simulation and introduce a corpus of 10,489 nationally aired television ads with 14,164 hand-collected annotations to show that uncorrected LLM-label coefficients sometimes land closer to a reliability-adjusted proxy for the construct than the human-label coefficients do, particularly on the features where human raters disagree most. These cases are difficult to identify using standard validation metrics, and the low human reliability that creates this possibility also makes correction difficult. We develop a diagnostic framework that treats both LLM and human labels as noisy measurement systems, along with an instrumental-variable estimator that identifies the underlying-construct coefficient, under the stated independence and exclusion assumptions, when a second independent rater on outcome-linked observations is available, at validation budgets that scale with human reliability.
Document Type
Article
Keywords
advertising, large-language models, multimodal AI, ad creative, generative AI, LLM annotation, generated regressors, measurement error, interrater reliability, construct validity, instrumental variables
Disciplines
Marketing
Source
SMU Cox: Marketing (Topic)
Language
English
