Micro Language Models Enable Instant Responses

Wen Cheng¹ Tuochao Chen¹ Karim Helwani² Sriram Srinivasan² Luke Zettlemoyer¹ Shyamnath Gollakota¹

¹Paul G. Allen School of Computer Science & Engineering, University of Washington ²Meta AI

EMNLP 2026 Main Conference (Oral)

Abstract

Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M–1B parameter language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro language models (µLMs): ultra-compact models (8M–30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency. We show that useful language generation survives at this extreme scale with our models matching several 70M–256M-class existing models. We design a collaborative generation framework that reframes the cloud model as a continuator rather than a respondent, achieving seamless mid-sentence handoffs and structured graceful recovery via three error correction methods when the local opener goes wrong. Empirical results show that µLMs can initiate responses that larger models complete seamlessly, demonstrating that orders-of-magnitude asymmetric collaboration is achievable and unlocking responsive AI for extremely resource-constrained devices.

Figure 1 showing the on-device micro language model initiating the response and the cloud LLM continuing it. — The on-device micro language model μLM initiates the response, which the cloud LLM continues.

Contributions

Extreme Scale Efficacy

We train decoder-only µLMs in the 8.8M–29.5M range and show that useful short response generation survives at this extreme scale.

Semantic Continuity

We reframe the cloud model as a continuator, enabling seamless mid-sentence handoff after a short on-device opener.

Graceful Error Recovery

We design three recovery modes—Explicit correction, Natural recovery, and Humor-aware recovery—to handle imperfect local prefixes.

Demo

Swen-28M generates the first 8 words, followed by the cloud model continuing from the prefix.

Model

Model checkpoint: Swen-28M

Citation

If our idea inspires your work, welcome cite:

@misc{cheng2026microlanguagemodelsenable,
  title        = {Micro Language Models Enable Instant Responses},
  author       = {Wen Cheng and Tuochao Chen and Karim Helwani and
                  Sriram Srinivasan and Luke Zettlemoyer and Shyamnath Gollakota},
  year         = {2026},
  eprint       = {2604.19642},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2604.19642}
}