Multitrait–Multimethod Validation of Human Skills Measurement via Authentic Student Voice

This is the central threat to validity for any language-based assessment, and no amount of agreement with human raters answers it. A model can reproduce trained raters almost perfectly and still be rewarding verbal fluency, because the human raters may have been rewarding verbal fluency too. Resolving it requires evidence from outside the scoring relationship.

Design. We used a multitrait–multimethod (MTMM) design (Campbell & Fiske, 1959), the standard approach to this problem. Three competencies were each measured by four independent methods, producing twelve measures in a fully crossed matrix. If the scores carry real construct signal, measures of the same competency by different methods should agree strongly, while measures of different competencies by the same method should agree only moderately. If the scores mostly carry method signal (fluency, halo, response style), the pattern inverts.

The three competencies were Grit, Curiosity, and Self-Regulation, chosen because theory predicts they are related but distinguishable, which makes discriminant validity a real test rather than a formality. The four methods were IMPACTER voice scoring, teacher rubric ratings, coded performance scenarios, and short-form published self-report scales.

Interpretation, stated conservatively. The eloquent-talker hypothesis predicts high method variance for the voice measures specifically: students who speak well scoring high on every competency. That is not what the data show. Voice measures carried less method variance (mean 6%) than teacher ratings (mean 12%), and voice measures of different competencies correlated less with each other (.33–.42) than each correlated with an independent measure of the same competency (.71–.76). The system differentiates between competencies rather than scoring articulateness.

What this study does not establish: that scores predict behavior or outcomes outside the assessment; that scores are stable over time; or that the instrument is appropriate for high-stakes individual decisions. Those require separate evidence, and §7 states what is outstanding.

IMPACTER's production scoring suite reports agreement with trained human raters (quadratic weighted kappa, exact accuracy, adjacent accuracy), evaluated against the thresholds used for automated scoring of constructed response (Williamson, Xi, & Breyer, 2012). Those figures establish something real and narrow: the engine reproduces the judgments of trained raters applying a published rubric.

They do not establish construct validity, and it is worth being precise about why. Agreement is a relationship between two raters. If both raters are influenced by the same irrelevant feature, such as length, vocabulary, or fluency, they will agree with each other while both being wrong about the competency. High agreement is therefore consistent with both a valid instrument and an invalid one. The literature on automated scoring has been explicit on this point: criterion agreement is necessary and insufficient (Yan, Rupp, & Foltz, 2020).

Construct validity requires evidence that the assessment elicits and captures the psychological processes theoretically associated with the construct (Messick, 1995; AERA, APA, & NCME, 2014). The MTMM design supplies that evidence by introducing methods that share no machinery with the scoring engine: a teacher watching a student over a semester, a coder reading a vignette response, a student answering Likert items. Those methods have their own biases, but they are different biases. Where independent methods with non-overlapping error converge, the shared signal is the construct.

All three competencies support goal-directed learning and share developmental precursors in early self-control, so they should correlate positively. But they diverge in mechanism: grit emphasizes stamina where curiosity may fade once a question is answered; curiosity motivates exploration where self-regulation often requires constraining it; self-regulation involves inhibition where grit involves sustained activation. The prediction is therefore moderate positive inter-trait correlation: meaningfully above zero, meaningfully below the same-trait correlations.

This prediction is what makes the design falsifiable. Uniformly high correlations across all three competencies within the voice method would indicate the model was scoring a single general quality of response. Near-zero inter-trait correlations would indicate the constructs had been defined into artificial independence.

Example prompts. Grit: "Tell me about a time you worked on something that took a long time to finish. What kept you going when it got hard?" Curiosity: "Describe a time when you didn't understand something and wanted to figure it out. What did you do?" Self-Regulation: "Tell me about a situation where you had to control your emotions or focus even though it was difficult."

Audio was transcribed by automated speech recognition and scored by a fine-tuned DeBERTa-v3-base encoder with a CORAL ordinal regression head, against construct-specific rubrics on a 0–4 scale (0–1 Foundational, 2 Emerging, 3 Developing, 4 Competent/Advanced). Scores were normalized for response length and calibrated against rotating anchor sets of expert-validated exemplars.

All measures were standardized within method. Outliers beyond 3.5 SD were winsorized (0.4% of values). Missing data (6.2% overall) were handled by full information maximum likelihood in the structural models and pairwise deletion in the correlation matrix; multiple imputation (20 imputations) produced substantively identical results.

We estimated a correlated-trait, correlated-method (CT-CM) confirmatory factor model (Eid et al., 2008): three correlated trait factors, four correlated method factors, twelve observed indicators, cross-loadings fixed at zero. This specification permits direct estimation of trait, method, and residual variance components. Fit was evaluated against conventional thresholds (CFI and TLI ≥ .95 excellent; RMSEA ≤ .05; SRMR ≤ .05). Beyond classical MTMM patterns we computed average variance extracted (Fornell & Larcker, 1981) and heterotrait–monotrait ratios (Henseler et al., 2015), which are stricter tests of discriminant validity. Measurement invariance was tested in the configural → metric → scalar sequence, with ΔCFI < .01 as the criterion (Cheung & Rensvold, 2002). Models were estimated in Mplus 8.8 with MLR estimation; descriptive and sensitivity analyses in R 4.3.1.

What this rules out. The eloquent-talker hypothesis makes a specific, testable prediction: if the voice measure primarily captures verbal facility, then method variance for the voice indicators should be high (a fluent student scores high on everything the instrument asks) and trait-specific variance should be low. The observed pattern is the reverse. Voice indicators carry mean method variance of .06 against mean trait variance of .62, a ratio of roughly 10:1, and their method variance is half that of teacher ratings (.12).

This is the finding that does the work in this report, and it deserves a conservative statement. It does not show the instrument is free of verbal-ability influence; language-based measurement will always carry some. It shows that the dominant source of variance in a score is which competency is being measured, not that the measurement happened through language. Method variance of 6% also sits well below the 25–50% commonly observed across social-science measurement (Podsakoff et al., 2003), which is a point in favor of standardized linguistic analysis relative to human observational rating rather than a claim of perfection.

On the English Learner result. This is the invariance test that matters most for an instrument built on student language, and it warrants care rather than celebration. Scalar invariance means the measurement model, including structure, loadings, and intercepts, operates equivalently for English Learners and non-English Learners in this sample. It does not mean the instrument is free of language-related advantage. It means that whatever advantage exists does not distort the relationship between the construct and its indicators. Item-level differential functioning analysis, which asks a narrower and harder question, has not been conducted and is listed in §7.3.

• Re-estimation on the current production release. This study scored under the pre-v6.0 release of the DeBERTa-v3-base suite. The architecture is unchanged in v6.0, so the findings are expected to replicate, but they should be re-estimated rather than assumed to carry forward.
• Criterion and predictive validity. Whether scores predict grade point average trajectory, course completion, attendance, disciplinary referral, or postsecondary enrollment. Not examined here.
• Test–retest stability. Deterministic scoring means an identical response receives an identical score. That is a property of the engine, not a stability estimate for the construct in students, and the two should not be conflated.
• Item-level differential functioning. Prompt-level DIF analysis across English Learner status, race/ethnicity, socioeconomic status, and disability status.
• Intervention sensitivity. Whether scores move in the expected direction following instruction, which is the evidence that makes these competencies assessable as malleable rather than fixed.
• Full discriminant coverage. Extension to the remaining five competencies, including the conceptually adjacent pairs.

Agreement with human raters establishes that the scoring engine reproduces trained raters. It does not establish that the instrument measures the competency it names. This study addressed the second question with the design built for it.

Result: variance in an IMPACTER score is dominated by which competency is being measured, not by the fact that the measurement was made through student language. Same-competency correlations across four independent methods averaged .72 against .31 across competencies; trait variance exceeded method variance by roughly ten to one; and the measurement model held equivalently across grade band, gender, and English Learner status.

Bounded to the release scored, the three competencies examined, and the California sample described in §4.1. Not a claim of readiness for high-stakes individual decisions. §7.3 is the complete list of what remains.