Heard The Same Way

Evaluated for bias?

Yes. Race and ethnicity, gender, income, native language (including Spanish) and learning differences: 27 subgroups, four audits.

Who conducted it?

Independent researchers at the University of Washington and Michigan State University (§4).

How often?

On every model version, before it reaches students. A failed check blocks the release. The next independent evaluation runs in Q1 2027, before Model 7.

1What we tested, and why

Bias in a language model does not always show up when you ask about it. It can sit in how a model responds to the way a person speaks, with the meaning held constant. A fair scorer has to be tested at that level, not only checked for overt bias.

Recent work established the method. By presenting the same content in African American English and in Mainstream American English, Hofmann et al. (2024) surfaced covert, dialect-triggered prejudice in language models that passed overt bias checks, and found that larger, preference-aligned models did not remove it. We used the same meaning-matched design on student responses, written and spoken (§3.2). Because nearly a quarter of the students in this sample speak Spanish at home, three of the four varieties we tested are ones Spanish-speaking students use: Chicano English, the English of students still learning it, and Spanish–English code-switching. We also tested individual grammatical features one at a time: the ones Pan et al. (2025) identify as driving model failure on dialect, and the Spanish-transfer constructions most common in our students’ writing (Table 4b).

Two further questions follow from the same literature. Spoken input raises the first: speech recognition makes more errors for Black speakers (Koenecke et al., 2020), so we measured whether those errors reach a student’s score (§3.3). The second is what bias should be measured against. People hold dialect prejudice too (Rickford & King, 2016), and teacher judgment is what voice scores supplement, so we asked whether voice widens or narrows the group gaps already present in teacher ratings of the same students (§3.6).

The core check is agreement with expert raters, group by group (§3.1). It is the comparison that surfaced subgroup departures in an operational essay scorer (Bridgeman, Trapani & Attali, 2012). It catches this kind of bias when it is there, which is why a clean result on it means something.

2Design

2.1 Sample and subgroups

The sample is the 8,612 students in grades 6–12 from 47 schools in 12 California districts studied in Report 2025-01, scored by the same release. Subgroup membership comes from district roster fields joined to de-identified records under each district’s data agreement. The model never sees a demographic field, and IMPACTER never infers one from a response. Home language comes from each district’s home language survey; Spanish is the largest group after English. Neurodivergence is taken from IEP primary disability category and Section 504 status. Groups under 100 students (†) are reported descriptively only. Non-binary students (‡, n = 102) enter the agreement analysis but are excluded from the multigroup models, as in Report 2025-01.

2.2 Four audits

Audit 1, human–machine agreement. Expert raters (inter-rater κ = .87; /iaa) scored the stratified subsample blind to demographics and to the machine score. For each subgroup we computed quadratic weighted kappa between raters and between rater and machine, and the standardized mean difference of machine minus human (Williamson, Xi & Breyer, 2012). This asks whether the machine departs from expert judgment more for some students than for others.

Audit 2, matched-guise dialect test. Following Hofmann et al. (2024), we drew 240 responses spanning every rubric level. Linguists who are fluent speakers of each variety rendered each one into African American English, Chicano English (Fought, 2003), the Spanish-influenced English of students learning English, and Spanish–English code-switched English, keeping every rubric-relevant idea. A second linguist confirmed meaning equivalence. Because any rewrite changes some wording, we also built a control: a meaning-preserving paraphrase in the same dialect. The control sets the noise floor, so a dialect effect is whatever the guises show beyond it. A spoken subset of 60 pairs per guise was recorded by fluent speakers and run through the full audio pipeline.

Audit 3, transcription. On 1,170 audio responses with human verbatim transcripts, we measured word error rate by subgroup. We then scored each response twice, from the ASR transcript and from the human transcript, so any transcription disparity that reached a student’s level would appear as an SMD (Koenecke et al., 2020).

Audit 4, differential functioning and invariance. DIF asks whether students with the same standing on a competency get different scores on a prompt because of group membership. The matching variable decides whether the test can see anything. Matching on the voice score would hide any bias common to every voice prompt, so we matched on a latent composite of the teacher, scenario, and self-report measures from Report 2025-01, which share no instrument with voice. We used ordinal logistic regression (Zumbo, 1999) on the 9 prompts for 18 focal groups, with Benjamini–Hochberg control (1995). We then extended the multigroup CFA of Report 2025-01 to the new dimensions.

2.3 What the model is asked to do

Covert dialect prejudice has been documented in general-purpose language models asked to judge the speaker: what they are like, what job they should hold, whether they are guilty (Hofmann et al., 2024). IMPACTER is not generative AI. It writes nothing, and it is never asked about the speaker. It is supervised machine learning: a DeBERTa-v3-base encoder (He et al., 2021) with an ordinal head, trained on responses that expert human raters scored against the rubric and on synthetic examples spanning every rubric level. It reads a response and returns a rubric level for the reasoning in it: what happened, what the student did, and what they learned.

That makes human agreement the definition of accuracy, not a side metric. Raters are trained until their own agreement reaches κ = .87, and the model is judged by quadratic weighted kappa against them, a measure of how closely its levels track expert human judgment. A model built this way can be no fairer than the raters it learns from, and it can only become less fair by matching them better for some students than for others. Audit 1 tests exactly that, and the dialect test checks whether the raters’ own judgments, as learned, carry a dialect penalty.

The scorer is a 184M-parameter encoder with no human-preference alignment layer. That matters because alignment has been shown to mask covert prejudice rather than remove it (Hofmann et al., 2024); here there is no such layer, so what the audits measure is what the model does. The narrower task is a reason to expect less bias, not proof of it. Pretraining associations live in the encoder whatever head sits on top. The audits test the expectation.

3Results

3.1 Agreement with expert raters

Overall, QWK_h–m = .85 against QWK_h–h = .87, and overall SMD = −.01. Every subgroup met all three criteria (Table 3). The largest departure was for students with a speech or language impairment (−.08), followed by English Learners and students with autism (−.06 each). All three are negative, and all three are groups whose spoken narration differs from the norm. The direction matters: where the machine departs from raters, it scores those students slightly below what raters gave them. We report it as the place to watch, and §5 makes it a monitored threshold.

3.2 The dialect test

If the encoder carried covert dialect associations (Hofmann et al., 2024) into rubric judgments, the African American English, Chicano English and Spanish-influenced guises would score lower than the MAE version of the same reasoning. Against the control’s 92.9% same-level rate, the written guises held the same level 91.2%, 92.5%, 90.4% and 88.8% of the time. The guise closest to a Spanish-speaking English learner’s own writing moved no more than the others, and no Spanish-transfer feature on its own moved scores beyond the noise floor (Table 4b). Downward moves outnumbered upward ones in every guise, which is the pattern to report rather than average away, but every standardized shift stayed inside |Δ| < .10 and no pair moved two levels. Spoken guises, which pass through transcription first, moved slightly more, as Audit 3 predicts.

3.3 What reaches the scorer

A spoken response is converted to text once, at the start. From then on the scorer has no audio. It has no acoustic model, no voiceprint, and no measure of pronunciation, accent, fluency or tone. It does not grade vocabulary or grammar either. It reads the architecture of the reasoning: temporal sequence, causal links, and connection to others. This is the relevant point in Koenecke et al. (2020), who found commercial speech recognition made nearly twice as many word errors for Black speakers and traced the gap to acoustic models. The scorer contains no acoustic model.

The conversion step itself is not exempt, so we measured it. Word error is higher for Black students (10.4% against 6.1% for White students), and higher for students whose home language is Spanish (9.1%), higher still for English Learners and for students with a speech or language impairment. The question that matters is whether those errors changed anyone’s level. Scoring the same audio from the machine transcript and from a human verbatim transcript, the difference stayed at or under −.06 SD in every group. Misheard words rarely break the structure being read. Typed responses remain available to any student.

3.4 Differential functioning

Matched on the external criterion, no prompt showed moderate or large DIF for any focal group. The largest effect, .027 for students with a speech or language impairment, is still Category A. The ordering repeats Table 3: the groups whose narration differs most sit closest to the line, while staying under it.

3.5 Measurement invariance

Scalar invariance, already shown for grade band, gender and English Learner status, held across race and ethnicity, home language (Spanish vs English), meal eligibility, and IEP/504 status. The same structure, loadings and intercepts operate in every group, which is what licenses comparing group means at all.

3.6 Does voice amplify existing gaps?

Group differences in a measured skill can be real, and the Standards separate such impact from bias (AERA, APA & NCME, 2014, ch. 3). The test that answers the word amplify is comparative. Human judgment is the baseline, and it is not neutral: listeners penalize dialect too (Rickford & King, 2016), and in Report 2025-01 teacher ratings carried five times the method variance of voice. If a gap is larger on voice scores than on teacher ratings of the same competency, voice is adding it. In every focal group the voice gap was the smaller one, by a mean of −.09 SD. For students whose home language is Spanish, teacher ratings showed a gap of −.18 and voice −.07. The largest reductions were for students with IEPs and for English Learners, the students whose teachers’ impressions are most likely to be shaped by how they talk.

4Who ran it, and how often

• Independent evaluation. The study was conducted independently by researchers at the University of Washington (Jerrod Robert, Ph.D.; Sungmin Pak, Ph.D.) and Michigan State University (Derek Seji, Ph.D.; Michelle Greenblat, Ph.D.). IMPACTER provided de-identified data and model documentation.
• Every model version. The full evaluation, including the dialect test, repeats on every model version before it reaches students. A failed criterion in Table 2 blocks the release. The next independent evaluation runs in Q1 2027, before Model 7 reaches students, and this report will be reissued with its results.
• On request. Any partner district can have Tables 3, 4 and 8 computed for its own students, including its Spanish-speaking students, reported only to the district.

5Limits

Grades. The sample is grades 6–12. Elementary students, including districts serving TK–8, are not represented. Neurodivergence is a records proxy that misses undiagnosed students. Dialects: four varieties were tested, three of them used by Spanish-speaking students; Vietnamese- and Tagalog-influenced English, Hawaiian Creole, and Indigenous Mexican languages were not. Responses written entirely in Spanish are outside this report. Small groups: American Indian or Alaska Native and Native Hawaiian or Pacific Islander students fell below the inferential floor. For them, no flag means too little data to test, not evidence of fairness. Watch list: speech or language impairment, English Learners and autism sit closest to every criterion. They get a tighter internal threshold (|SMD| < .08) at each release.

·Acknowledgments

D. Hwang, A. Michelson, A. Le and D. Siddiqi of Impacter Education Corporation provided de-identified data and model documentation for this evaluation.

·References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. AERA.
2. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society B, 57(1), 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x
3. Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1), 27–40. doi:10.1080/08957347.2012.635502
4. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. doi:10.1080/10705510701301834
5. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. doi:10.1207/S15328007SEM0902_5
6. Fought, C. (2003). Chicano English in context. Palgrave Macmillan.
7. He, P., Gao, J., & Chen, W. (2021). DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv:2111.09543.
8. Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). AI generates covertly racist decisions about people based on their dialect. Nature, 633, 147–154. doi:10.1038/s41586-024-07856-5
9. Hwang, D., Michelson, A., Le, A., & Siddiqi, D. (2025). Multitrait–multimethod validation of human skills measurement via authentic student voice (Technical Report 2025-01). IMPACTER Education Corporation.
10. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349. doi:10.1207/S15324818AME1404_2
11. Johnson, M. S., Liu, X., & McCaffrey, D. F. (2022). Psychometric methods to evaluate measurement and algorithmic bias in automated scoring. Journal of Educational Measurement, 59(3), 338–361. doi:10.1111/jedm.12335
12. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020). Racial disparities in automated speech recognition. PNAS, 117(14), 7684–7689. doi:10.1073/pnas.1915768117
13. Loukina, A., Madnani, N., & Zechner, K. (2019). The many dimensions of algorithmic fairness in educational applications. Proceedings of the 14th Workshop on Innovative Use of NLP for Building Educational Applications, 1–10. doi:10.18653/v1/W19-4401
14. Pan, E., Choi, A. S. G., ter Hoeve, M., Seto, S., & Koenecke, A. (2025). Analyzing dialectical biases in LLMs for knowledge and reasoning benchmarks. Findings of EMNLP 2025. arXiv:2510.00962.
15. Rickford, J. R., & King, S. (2016). Language and linguistics on trial: Hearing Rachel Jeantel (and other vernacular speakers) in the courtroom and beyond. Language, 92(4), 948–988. doi:10.1353/lan.2016.0078
16. Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. doi:10.1111/j.1745-3992.2011.00223.x
17. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Department of National Defense, Ottawa.

Report 2026-01 · v1.0 · Sample of Report 2025-01 · Scoring release DeBERTa-v3-base, CORAL ordinal head · Independent evaluation by researchers at the University of Washington and Michigan State University · Companion: Technical Report 2025-01, Model Card v6.0 · info@impacterpathway.com