FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
A new benchmark and evaluation framework, FAIRLENS, tests whether vision-language models make fair and valid decisions in hiring, legal, and healthcare contexts. Using over 100,000 image-question pairs per model built from real face images across gender, race, and age groups, the study evaluates eight VLMs through four complementary lenses. The dominant failure, it finds, is unwarranted inference rather than unequal treatment: models infer qualifications, threat, illness, or professional roles from faces instead of abstaining, with the weakest model doing so on 99% of the question-image pairs that do not support an answer. The authors warn that parity metrics alone miss these harms, and that bias in free-text generation is only loosely tied to structured-answer accuracy. Fair high-stakes behavior, they argue, requires both similar treatment across groups and a willingness to refuse inference from appearance.
-->
[Submitted on 1 Sep 2026]
Title:FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
View a PDF of the paper titled FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making, by Vahid Reza Khazaie and 2 other authors
View PDF HTML (experimental)
Abstract:Vision-language models (VLMs) are increasingly used to make decisions from visual inputs. We introduce FAIRLENS, a benchmark and evaluation framework for measuring both the fairness and the validity of VLM responses in three high-stakes domains: hiring, legal, and healthcare. FAIRLENS pairs real face images spanning gender, race, and age groups with closed- and open-ended questions, giving more than 100K image-question pairs per model, and evaluates responses from four complementary views: demographic parity over adverse outcome rates, soundness, demographic association over unsupported roles and statuses, and bias in free-text generation. Soundness is the central validity criterion: a response is sound when it follows the evidence stated in the question and abstains when the image cannot support an answer. Evaluating eight VLMs, we find that the primary failure is unwarranted inference rather than unequal treatment. Models routinely infer qualifications, threat, illness, or professional role from a face instead of abstaining, and the weakest model does so on 99% of the questions its input cannot answer. These failures are most severe in legal and healthcare, where recognizing insufficient evidence matters most, and disparity metrics alone would miss them: parity gaps are small in absolute terms, yet when baseline adverse rates are low the same gap means one demographic group receives adverse labels several times as often as another, and a small gap can equally reflect a model that treats every group unsafely. Bias in free-text responses is only loosely coupled to multiple-choice accuracy, so correct structured answers do not imply safe generation. FAIRLENS shows that fair high-stakes VLM behavior requires similar treatment across groups and refusal to infer high-stakes attributes from appearance, and its question suite transfers to any face corpus with demographic annotations.
Comments: Code and benchmark resources are available at this https URL
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2609.01691 [cs.CV]
(or arXiv:2609.01691v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.01691
arXiv-issued DOI via DataCite
Submission history
From: Vahid Reza Khazaie [view email] [v1] Tue, 1 Sep 2026 16:06:50 UTC (196 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making, by Vahid Reza Khazaie and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new | recent | 2026-09
Change to browse by:
cs cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)