2026-07-10 04:00 UTCOriginal source2 min readUpdated: 2026-07-10 08:22 UTC

Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

This research proposes a cost-efficient human-LLM collaborative annotation framework to construct multilingual stereotype datasets. Applied to Spanish, it yields EspanStereo, covering multiple Spanish-speaking countries. Evaluations show significant variation in LLM stereotypical behavior across countries, highlighting the need for culturally grounded assessments.

SourcearXiv Computational LinguisticsAuthor: Weicheng Ma, John Guerrerio, Soroush Vosoughi

Article intelligence

InvestorsAdvanced

Key points

Proposes a human-LLM collaborative framework that combines LLM-generated candidate stereotypes with in-culture annotator validation.
Constructs EspanStereo, the first Spanish stereotype dataset spanning multiple countries, capturing both documented and culturally specific biases.
Evaluation of Spanish LLMs reveals country-dependent stereotype behavior, emphasizing the importance of culturally specific assessment.
Framework is adaptable to other languages, paving the way for multilingual stereotype benchmarks.

Why it matters

This matters because proposes a human-LLM collaborative framework that combines LLM-generated candidate stereotypes with in-culture annotator validation.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 8 Jul 2026]

Title:Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

View a PDF of the paper titled Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration, by Weicheng Ma and 2 other authors

View PDF HTML (experimental)

Abstract:Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework's effectiveness in identifying nuanced, region-specific biases. Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.

Comments: Weicheng Ma, John Guerrerio: equal contribution; published in EMNLP 2025 Main

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2607.07895 [cs.CL]

(or arXiv:2607.07895v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.07895

arXiv-issued DOI via DataCite (pending registration)

Related DOI:

https://doi.org/10.18653/v1/2025.emnlp-main.1221

DOI(s) linking to related resources

Submission history

From: John Guerrerio [view email] [v1] Wed, 8 Jul 2026 20:06:00 UTC (8,186 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration, by Weicheng Ma and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)