Skip to content
AI News HubLIVE
Original source2 min read

TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

Summary

TalkFa introduces a unified Farsi dialogue benchmark with three complementary datasets reviewed by native speakers. Experiments show LoRA recovers most gains with 25-50% of data, automatic metrics overestimate quality, and frontier LLMs still struggle zero-shot.

SourcearXiv Computational LinguisticsAuthor: Neda Jamshidi, Kamyar Zeinalipour, Fahimeh Akbari, Monica Bianchini, Marco Maggini, Marco Gori
TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 1 Sep 2026]

Title:TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

View a PDF of the paper titled TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding, by Neda Jamshidi and 5 other authors

View PDF HTML (experimental)

Abstract:Farsi, spoken by more than 120 million people, lacks a comprehensive benchmark for dialogue generation and understanding. We introduce TALKFA, a unified benchmark comprising three complementary datasets: (1) WIKI-FADIAL, 4.2K Wikipedia-grounded dialogues for knowledge-grounded generation; (2) DAILYDIALOG-FA, 6.6K dialogues annotated for dialogue acts and emotions; and (3) PLAYDIAL-FA, 2.1K theatrical dialogues with sentiment labels. While LLMs assist data construction, every dialogue undergoes multi-stage review and revision by native Farsi speakers, and only the final human-approved dialogues are released. Experiments with six LLAMA and MISTRAL models show that LoRA substantially improves dialogue generation while requiring only 25-50% of the training data to recover over 90% of the final performance gains. Across classification tasks, FABERT achieves the best dialogue-act performance, LORA-MISTRAL-7B performs best on emotion recognition, and MISTRAL-24B achieves the highest sentiment score. Human evaluation and independent external validation demonstrate the reliability of the benchmark, while comparisons with GPT-4.1 as an LLM judge reveal that automatic metrics substantially overestimate dialogue quality. Zero-shot evaluation with frontier LLMs further shows that TalkFa remains a challenging benchmark. We will release all datasets, annotation guidelines, code, and checkpoints.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2609.01810 [cs.CL]

(or arXiv:2609.01810v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.01810

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Neda Jamshidi [view email] [v1] Tue, 1 Sep 2026 19:35:25 UTC (1,460 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding, by Neda Jamshidi and 5 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-09

Change to browse by:

cs

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

InvestorsAdvanced

Key points

  • TalkFa combines Wiki-FaDial, DailyDialog-Fa, and PlayDial-Fa, totaling about 12.9K native-speaker-reviewed Farsi dialogues.
  • LoRA fine-tuning needs only 25-50% of training data to recover over 90% of final performance gains.
  • FABERT leads dialogue-act classification, LoRA-Mistral-7B leads emotion recognition, and Mistral-24B leads sentiment scoring.
  • GPT-4.1-as-judge comparisons show automatic metrics substantially overestimate dialogue quality, and zero-shot frontier LLMs remain challenged.

Highlights and analysis are generated automatically and may contain errors. Check the original source.