Skip to content
AI News HubLIVE
Original source2 min read

MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

Summary

Researchers propose MIST, a synthetic multi-turn voice-driven code generation dataset for IoT devices. Experiments reveal a significant gap between open- and closed-weight multimodal LLMs, with even frontier closed models having substantial room for improvement. The dataset and extensible generation framework are released to facilitate research on mixed-initiative voice assistants that reason about physical world constraints.

SourcearXiv Computational LinguisticsAuthor: Maximillian Chen, Xuanming Zhang, Michael Peng, Zhou Yu, Alexandros Papangelis, Yohan Jo
MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[2605.06897] MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

[Submitted on 7 May 2026]

Title:MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

View a PDF of the paper titled MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes, by Maximillian Chen and Xuanming Zhang and Michael Peng and Zhou Yu and Alexandros Papangelis and Yohan Jo

View PDF HTML (experimental)

Abstract:The rise of Internet of Things (IoT) devices in the physical world necessitates voice-based interfaces capable of handling complex user experiences. While modern Large Language Models (LLMs) already demonstrate strong tool-usage capabilities, modeling real-world IoT devices presents a difficult, understudied challenge which combines modeling spatiotemporal constraints with speech inputs, dynamic state tracking, and mixed-initiative interaction patterns. We introduce MIST (the Multimodal Interactive Speech-based Tool-calling Dataset), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices. We find that there is a significant gap between open- and closed-weight multimodal LLMs on MIST, and that even frontier closed-weight LLMs have substantial headroom. We release MIST and an extensible data generation framework to build related datasets in order to facilitate research on mixed-initiative voice assistants which reason about physical world constraints.

Comments: Project Page: this https URL

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)

Cite as: arXiv:2605.06897 [cs.CL]

(or arXiv:2605.06897v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.06897

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Maximillian Chen [view email] [v1] Thu, 7 May 2026 19:57:39 UTC (3,425 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes, by Maximillian Chen and Xuanming Zhang and Michael Peng and Zhou Yu and Alexandros Papangelis and Yohan Jo

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

cs cs.AI cs.HC cs.MM cs.SD eess eess.AS

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • MIST is a voice-driven code generation task for smart home IoT devices, simulating mixed-initiative multi-turn conversations.
  • A notable performance gap exists between open- and closed-weight multimodal LLMs on MIST, with closed models still far from perfect.
  • The dataset and an extensible generation framework are open-sourced to advance research on IoT-aware voice assistants.
  • This work fills a research gap in combining spatiotemporal constraints, dynamic state tracking, and mixed-initiative patterns in voice interaction.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models