---
title: "Training full-duplex LLMs for spoken dialogue"
slug: full-duplex-llm-training
question: "How are full-duplex (simultaneous bidirectional speech) LLMs trained, and which architectures, data, and training methods achieve low-latency, interruptible spoken dialogue?"
status: published
depth: deep
created: 2026-08-07
updated: 2026-08-07
summary: "Full-duplex spoken dialogue models — systems that listen and speak at the same time — became a distinct research field between 2023 and 2026, moving from the first speech-in/speech-out language models to a crowded design space of dual-stream transformers, frozen-LLM adapters, and single-stream 'native duplex' recipes. The evidence shows training data that contains genuine overlapping speech is the binding constraint, not architecture; latency and interruption handling are now measurable but benchmarks disagree about what to measure; and post-training alignment (RL/DPO on turn-taking behavior) is the fastest-moving lever. Confidence is moderate: the field is young, most frontier systems are preprints, and no standardized evaluation exists yet."
disciplines: ["machine learning", "speech processing", "natural language processing"]
tags: ["full-duplex speech", "spoken dialogue systems", "speech language models", "turn-taking", "interruption handling", "audio codecs"]
source_count: 111
year_range: [2023, 2026]
confidence: moderate
search:
  databases: ["OpenAlex", "arXiv", "Semantic Scholar", "Crossref"]
  queries:
    - "full-duplex speech"
    - "full duplex speech LLM"
    - "full-duplex dialogue"
    - "full duplex spoken dialogue system"
    - "duplex voice conversation"
    - "simultaneous speech interaction large language model"
    - "listening while speaking"
    - "barge-in spoken dialogue"
    - "interruption handling spoken dialogue LLM"
    - "streaming speech-to-speech dialogue"
    - "turn-taking spoken dialogue large language model"
    - "speech-text interleaved training"
    - "speech language model dialogue"
    - "omni-modal large language model speech"
    - "audio codec language model dialogue"
    - "real-time voice interaction LLM"
    - "realtime speech LLM dialogue"
    - "spoken dialogue system large language model"
    - "voice assistant streaming dialogue"
    - "speech LLM survey"
  last_run: 2026-08-07
---


## Summary

Between 2023 and 2026, "full-duplex" went from a radio-engineering term to a research program for spoken AI: language models that can listen and speak simultaneously, be interrupted mid-sentence, and interrupt in turn. This review synthesises 110 retrieved sources — 59 peer-reviewed papers (NeurIPS, ICML, ICLR, AAAI, ACL/EMNLP/NAACL, Interspeech, ICASSP, ASRU, TACL) and 51 preprints or technical reports — on how such models are trained. Three findings carry the review. First, the binding constraint is data, not architecture: models trained on corpora containing real overlapping speech (Fisher-scale, thousands of hours) behave markedly more duplex than models fine-tuned only on clean turn-taking data, and several 2025–2026 papers argue this explicitly. Second, there is a genuine fork between *engineered* turn-taking (external VAD/endpointer modules, state machines, semantic end-pointers) and *learned* turn-taking (dual-stream or joint-token objectives plus RL/DPO alignment); both work, and the strongest 2026 results combine them. Third, evaluation is the field's weak joint: at least a dozen benchmarks define latency, takeover, and overlap metrics differently, several are built on synthetic or scripted interruptions, and independent numbers for commercial systems come only from third-party benchmarks. Confidence in the overall picture is moderate — the field is young, half the corpus is preprints, and no single result has been independently replicated at scale.

## Why this question

Human conversation is not turn-based: speakers overlap, backchannel, interrupt, and finish each other's sentences, coordinated through prosody, gaze, and gesture within ~200 ms [59]. Every spoken interface that imposes rigid turn-taking — user stops, system thinks, system speaks — pays a latency tax that users experience as "why is my agent so slow" [58]. The commercial push (GPT-4o's real-time voice mode [108], and its successors) made naturalistic, interruptible voice the expected interaction paradigm, and the open-source response was a flood of "omni" and "duplex" models from 2024 onward.

What turns on the answer is practical: whether a team building a voice agent should train a purpose-built duplex model end-to-end, bolt a duplex layer onto an existing LLM, or assemble a cascaded pipeline with a good end-pointer. The three options differ by orders of magnitude in data and compute. This review asks what the peer-reviewed and preprint evidence actually supports for each choice, and where the evidence stops.

## Scope and methods

**Question.** How are full-duplex LLMs — speech-in/speech-out language models that can process user audio while generating their own speech — trained? Included are architecture choices (tokenizers, stream design, fusion), training data (real vs synthetic, overlapping vs turn-based), training objectives and stages (pretraining, SFT, preference/RL alignment), and the components of duplex behaviour (turn-taking, barge-in/interruption handling, backchannels, latency). Evaluation methodology is included insofar as it is what training decisions are measured against.

**Inclusion criteria.** Works published or posted 2023-01-01 to 2026-08-07; spoken-dialogue language models (speech-in speech-out or omni-modal with duplex claims), their components (codecs, end-pointers, turn-taking predictors), benchmarks, and surveys in the full-duplex space; venues of record for ML/speech (NeurIPS, ICML, ICLR, AAAI, ACL, EMNLP, NAACL, Interspeech, ICASSP, ASRU, SLT, TACL, LREC-COLING, SIGDIAL, ICMI) plus arXiv preprints, which is where most of this field actually lives. **Exclusion criteria.** Pure ASR, TTS, or speech understanding with no dialogue component; half-duplex spoken dialogue systems that never address simultaneity (e.g., classic turn-based task-oriented systems, unless used as training-data or baseline evidence); simultaneous *translation* (a different problem); non-LLM classic spoken dialogue systems; papers unreachable through any database or archive (several prominent models rumoured in the field, e.g., "FullDuplex" by Hume AI, could not be retrieved in any indexed source and are deliberately not cited).

**Databases and dates.** OpenAlex, arXiv, Semantic Scholar (best-effort; frequently rate-limited), Crossref (DOI resolution, venue verification, retraction checks). Searches run 2026-08-07. Twenty query strings (listed in frontmatter) × 3 databases, plus snowballing: citation-graph follow-ups from anchor papers (Moshi, Mini-Omni, LLaMA-Omni, Freeze-Omni, NTPP, SALM-Duplex, SALMONN-omni, Full-Duplex-Bench, the two full-duplex surveys) and targeted retrieval of ~20 suspected-anchor works by exact title across OpenAlex, Crossref, and arXiv (all of which failed for the unretrievable items above — evidence that several widely discussed "papers" circulate only outside indexed archives).

**Screening counts.** 1,454 raw records retrieved; 1,118 after DOI/title deduplication (336 duplicates removed); 363 after a keyword screen on title (755 excluded as off-topic — wireless full-duplex, speech pathology, education, emotion, etc.); 110 included after manual relevance screening (253 excluded: out of scope, or duplicate versions); all 110 were then read — 78 in full text (arXiv HTML/ar5iv, open-access proceedings, publisher pages) and 32 at abstract level (paywalled IEEE/ACM/MIT-Press items without open preprints; flagged as `abstract-only` in the evidence table and hedged accordingly in the text). Every DOI was resolved from an API response this session; every load-bearing source was checked for retraction via Crossref `update-to` (none found; the arXiv `10.48550` DOIs are not registered at Crossref, so those were checked by direct archive access). Venues were verified per-record against Crossref rather than trusted from secondary metadata; where a paper's arXiv "comments" field or Crossref record named a conference, the conference version is cited.

**What this review does not cover.** Sub-second streaming *text-to-speech* quality engineering (CosyVoice-style streaming TTS appears only as a component [107]); multimodal (vision+speech) evaluation beyond the omni models included; and the acoustic echo-cancellation/signal-processing literature beyond one ICASSP 2025 representative [24].

## The landscape

The literature has a clear chronology and an equally clear shape: it is young, exploding, and preprint-dominated. Before 2023 there was essentially nothing: spoken dialogue existed as cascaded ASR→NLU→TTS pipelines, and the only generative spoken-dialogue language models were dGSLM [32] and SpeechGPT [31] — the first speech-in/speech-out LLMs, both strictly turn-based, with SpeechGPT's chain-of-modality incurring multi-second latency. 2023 also supplied the enabling ingredients: text-pretrained speech LMs showing the value of warm-starting from text (TWIST) [5], scaling analyses concluding that speech-only LMs learn language far slower than text LMs [33], and the audio-perception side (AudioPaLM [89], SALMONN [91], LTA [90], Qwen-Audio/Qwen2-Audio [92]) that later became the "listening" half of duplex models.

The field proper begins in October 2024 with Moshi [1], the first open model with a genuinely dual-stream design — one transformer generates audio and text tokens while a second consumes the user's stream in parallel — and with Mini-Omni [61], which showed a single 0.5B model could stream audio in and out. GPT-4o's system card [108] documented the commercial reference point — a speech-to-speech model that handles audio as text tokens — though the card itself reports no latency figures, and independent latency and interruption numbers for commercial systems come only from third-party benchmarks [98] [10] [101]. From there the space bifurcated. One line froze the text LLM and added speech modules (Freeze-Omni [63], MinMo's full-duplex predictor [66], FlexDuo [86]); another kept a single autoregressive model and interleaved everything into one token stream (OmniFlatten [14], NTPP [7], BayLing-Duplex [71]); a third pushed toward omni-modal real-time systems with a separated "thinker" and "talker" (Qwen2.5-Omni [64], Qwen3-Omni [65], VITA-1.5 [2], MiniCPM-o 4.5 [70], GLM-4-Voice [67], Baichuan-Omni-1.5 [69], VITA [68]). Meanwhile 2025–2026 produced a striking volume of *evaluation* work — at least thirteen full-duplex benchmarks — and three dedicated surveys: two on full-duplex specifically [104] [105], one on speech language models broadly (ACL 2025) [42], and an earlier spoken-dialogue-model survey [106] — the usual sign of a field trying to find its feet.

Three shape observations matter. First, one era dominates: nearly everything is from 2024–2026, and the 2026 output (in this session's window) is already the largest single year — the field is mid-explosion. Second, the anchor systems are a small set: Moshi (and its fine-tunes) appears as the baseline in almost every benchmark [22] [98] [48] [100] [101] [17] [18], so many "independent" evaluations share one reference implementation. Third, the surveys themselves disagree about the right taxonomy — one organises systems by whether synchronisation is engineered or learned [104], the other by an architectural hierarchy from cascaded (L0) to native single-model (L3) [105] — which is itself evidence that the field has not settled its own ontology.

## Theme 1 — The token layer: what the model actually models

Every full-duplex speech LM is a language model over *some* discretisation of audio, and the choice of codec sets the ceiling on both quality and latency. The dominant design is a residual-vector-quantised (RVQ) codec producing acoustic tokens at 12.5–25 Hz per codebook, consumed either as a flattened sequence or as parallel codebook channels. Moshi uses Mimi, a 12.5 Hz, 8-codebook codec with a token pattern that delays codebook stacking to keep latency at 160 ms theoretical / ~200 ms practical [1]; Baichuan-Omni-1.5 uses an 8-layer RVQ tokenizer at 12.5 Hz [69]; WavTokenizer pushes to a single codebook at 40–75 tokens/s [9]; SALM-Duplex operates at a 0.6 kbps codec rate [15].

The recurring finding is that codec *semantics* matter more than bitrate. An AAAI 2025 study showed codec tokens carry limited semantic content and proposed enriching codec training with semantic objectives [13]; follow-up work demonstrated that discrete-token inconsistency across RVQ layers causes generation errors (omissions, repetitions) and that mitigation requires careful token-level objectives [43]; a NAACL-SRW study argued codec and LM should be co-designed rather than bolted together [111]; and LLM-Codec (ACL 2026) trains the codec itself with language-model objectives [55]. For *duplex* specifically, low bitrate is not just efficiency: fewer tokens per second means fewer autoregressive steps per utterance, which is why SALM-Duplex and WavTokenizer-style codecs underpin low-latency designs. Multi-token prediction (VocalNet, EMNLP 2025) attacks the same wall from the decoding side, generating several tokens per step for speed without quality loss [44].

A second, minority architecture avoids tokens entirely: SALMONN-omni (NeurIPS 2025) integrates a continuous speech encoder and decoder directly into the LLM without codec injection, reporting gains over codec-based duplex baselines and claiming the codec bottleneck is the main thing standing between speech LMs and full-duplex quality [3]. The trade-off is standard engineering: continuous features avoid discretisation loss but lose the clean autoregressive interface that makes token-stream duplex designs (Moshi, NTPP) simple to train.

## Theme 2 — Architecture families: where the two streams live

Full-duplex requires the model to attend to the user's ongoing stream *while* producing its own output. The earliest LLM-based scheme (NeurIPS 2024) coupled an 8B LLM with a two-state neural finite-state machine that emitted control tokens to arbitrate speaking and listening [4]. The evidence since then supports four architectural families, with real performance differences but no settled winner.

**Dual-stream transformers.** Moshi's two-stream design — a 7B text backbone plus a 1.2B temporal transformer, one stream for self (speech+text) and one for the user — remains the canonical open implementation [1]. SyncLLM (EMNLP 2024) formalises the idea for text: generate while ingesting, with a synchronous objective and a large synthetic overlapping-text corpus, and reports large quality gains over dGSLM [35]. The "Duplex" model (EMNLP 2024) similarly re-frames dialogue as continuous-time token generation rather than turn exchange [34]. On the speech side, the Japanese J-Moshi adaptation fine-tunes Moshi for a new language by swapping the tokenizer and training on ~69k h of Japanese plus stereo data [16], and Human-1 does the same for Hindi with 26k h of real stereo conversations [87] — evidence that the dual-stream recipe transfers across languages. A controlled 2026 study asks how the user stream should be routed into the LLM at all: injecting it into the input channel gives better semantic grounding but corrupts the generation context under missed interruptions, whereas routing it through cross-attention preserves coherence at some cost to grounding [79]. A separate fusion line (LSLM, AAAI 2025) fuses a streaming self-supervised speech encoder with a TTS decoder mid-way, reporting 98% interruption-detection F1 and robust barge-in detection while the model is speaking [12].

**Single-stream interleaving.** LLaMA-Omni (ICLR 2025) showed a single decoder can ingest streaming speech and emit text plus speech with ~236 ms latency, trained on the open OmniInstruct set [11]; Mini-Omni2 extended the pattern to vision plus speech with duplex capabilities [62]. OmniFlatten flattens the entire audio-text dialogue history into one GPT sequence, generating speech and text tokens in a unified stream [14]. NTPP (ICML 2025) goes further with a *next-token-pair* objective over dual-channel audio — the model predicts both speakers' tokens jointly, which the authors argue matches the true joint distribution of conversation; they report better naturalness than Moshi at sub-220 ms latency using a single KV cache [7]. BayLing-Duplex shows a turn-based SpeechLM (GLM-4-Voice) can be converted to native duplex purely by re-serialising the sequence into interleaved user-speech/assistant-text/assistant-speech blocks with four dialogue-state tokens, then SFT + DPO — 92% turn-taking success, 100% interruption success on their eval, with no architectural change [71]. The "Listen-Write-Speak" recipe (Liberating LLM Capabilities) is a token-schema-only variant that keeps text generation as a visible first-class channel during duplex interaction [75].

**Frozen-LLM / modular duplex.** Task-oriented duplex systems add dialogue-state tracking on top of a Moshi-class model to get usable task behaviour [47]. Freeze-Omni freezes a 7B text LLM and trains lightweight speech-in/out adapters around it, reporting low-latency S2S dialogue with ~160–320 ms chunked processing [63]. MinMo (Alibaba) aligns an 8B multimodal model and then bolts on a separate Full-Duplex Predictor module that learns turn-taking from data; the paper reports ~600 ms theoretical / ~800 ms practical end-to-end latency [66]. FlexDuo provides a pluggable three-state (speak/idle/listen) control module that converts half-duplex speech dialogue systems into full-duplex ones, cutting false interruptions [86]. SoulX-Duplug replaces the VAD with a streaming *state-prediction* LLM (240 ms average latency) that unifies VAD, ASR, and turn detection [73]; UAF (2026) folds VAD, speaker recognition, ASR, and turn-taking detection into a single streaming sequence-prediction task with control state tokens, trained on simulated barge-in and noise [72]. DuplexMamba explores a linear-time Mamba backbone for duplex streaming [57], and the ICASSP 2026 "left-right brain" system — a modular multi-agent division of labour — won the HumDial full-duplex track, showing engineered modularity still competes [26].

**Omni-modal thinker-talker.** Qwen2.5-Omni introduced the Thinker-Talker pattern: the LLM ("thinker") generates text while a dual-track autoregressive "talker" converts its hidden states into audio tokens, avoiding a second full LLM forward pass [64]; Qwen3-Omni (2026) keeps the pattern with a MoE thinker and reports a theoretical 234 ms first-packet latency [65]. VITA-1.5 (NeurIPS 2025) shows a progressive three-stage training recipe — vision-language first, then audio input, then audio output — that avoids the modality-conflict problem and reaches near-real-time S2S without external ASR/TTS [2]. MiniCPM-o 4.5 (2026) pushes the pattern onto edge hardware: the LLM emits text at 3–4 tokens/s while a 0.3B decoder emits speech tokens, with GRPO post-training, running full-duplex in <12 GB RAM [70]. GLM-4-Voice generates text and speech token pairs with a dual-token loss [67]. The common thread across all four families: *someone* must decide when to yield the floor, and the literature splits on whether that decision is a learned prediction inside the model or an engineered module outside it.

## Theme 3 — Data: the binding constraint

The strongest and most consistent finding in this corpus is that full-duplex behaviour is learned from data that contains overlap, and such data is scarce. The 2026 survey of full-duplex systems makes the point explicitly: many architectures are *capable* of duplex operation, but observed behaviour is constrained by the interaction patterns present in training and evaluation data, and public corpora are far smaller than undisclosed industrial ones [105]. The 2025 survey reaches the same conclusion from the other direction, identifying synchronous data scarcity as a primary bottleneck [104].

The canonical real resource remains Fisher (2,000 h of telephone conversations, used to fine-tune Moshi [1] and dGSLM [32]) — a dataset recorded in 2004 and not designed for duplex training at all. The field's response has been threefold. First, *reuse and restore*: DuplexChat (2026) constructs ~415k h of speaker-separated two-speaker dialogue from web-scale podcasts via diarisation-guided extraction — the largest open resource by two orders of magnitude [96]; DialogueSidon restores degraded in-the-wild monaural dialogue into clean two-stream tracks [97]; pseudo-stereo data generated from single-speaker corpora was shown to improve dialogue speech generation [94]; and Sommelier (ACL 2026 industry) packages a scalable multi-turn pre-processing pipeline targeting overlap and back-channels specifically [51]. Second, *synthesise*: SpeechDialogueFactory generates high-quality scripted dialogue audio that matches human recordings on quality metrics [95]; the KE-Omni line scaled SFT to 60k+ h of synthetic speech dialogue [93]; InteractSpeech contributes a speech-dialogue interaction corpus [46]; PersonaPlex used 2,250 h of synthetic role- and voice-conditioned data to fine-tune Moshi [28]. Third, *collect natively*: Human-1's 26k h of real Hindi stereo conversations [87], J-Moshi's stereo data [16], Let's Go Real Talk's 340-hour face-to-face audio-visual MultiDialog corpus [36], and the HumDial challenge's real dual-channel recordings [29] [101]. Task-oriented spoken dialogue has its own resources — SpokenWOZ (NeurIPS 2023, 8k dialogues) [6] and RealTalk-CN [52] — and speaking-style-annotated dialogue data was shown to improve spoken LLM responses [37].

The disagreement is over whether synthetic data is good enough. SpeechDialogueFactory's own authors note their pipeline produces scripted turn-taking without genuine overlap dynamics [95]; Multi-Faceted Interactivity Alignment found that training on Fisher segments actually *degraded* a safety-tuned model's instruction following [76]; and the full-duplex survey argues real overlap patterns are what teach interruption behaviour [105]. The strongest counter-evidence for synthetic data is its cost-effectiveness — F-Actor (ACL 2026) trains the first open instruction-following duplex model on ~2,000 h of synthetic behaviour-annotated dialogue in two days on four A100s [49], and Slamming trains a speech LM on one GPU in a day [38]. Data scale also shows up in the content-quality gap: SOVA-Bench found cascaded systems retain ~59% knowledge-QA accuracy versus ~27% for the best open speech LLM [18], and RealTalk-CN's Chinese task-oriented benchmark finds the text pipeline still beats end-to-end speech models on chat quality [52] — suggesting that today's duplex training data improves *timing* faster than it improves *knowledge*.

## Theme 4 — Training recipes: stages, objectives, warm starts

Four training-recipe findings recur across independent papers.

**Text warm-starting is the single most reliable lever.** TWIST (NeurIPS 2023) showed text-pretrained speech LMs massively outperform cold-started ones [5]; scaling analyses explain why — speech-only LMs improve far slower with compute than text LLMs [33]; AudioPaLM demonstrated the speak-and-listen pattern on a text foundation [89]; and every 2024–2026 system reviewed here builds on a text LLM (Moshi on a 7B text model [1], Freeze-Omni [63], Qwen2.5-Omni [64], MiniCPM-o 4.5 on Qwen3-8B [70]). The open "full-duplex voice agent" paper (ASRU 2025) goes furthest: it claims *any* standard text LLM can be converted to duplex with streaming speech I/O and no extensive speech-text pretraining [21].

**Pretraining and data mixing avoid modality degradation.** Qwen3-Omni explicitly mixes unimodal and cross-modal data during early text pretraining and reports this avoids the modality-degradation failure (speech training eroding text ability) [65]; Liberating LLM Capabilities documents the same failure mode and fixes it with a text-first token schema [75]; VITA-1.5's progressive stage order (vision → audio-in → audio-out) is a third independent mitigation [2]. This is one of the few places where three papers from different labs converge on the same mechanism.

**The objective matters: joint vs conditional.** dGSLM's original turn-and-interruption tokens made interruption a *generated* event [32]; Moshi's dual-stream conditions generation on the user stream [1]; NTPP argues for modelling the joint distribution of both channels via next-token-pair prediction, and its naturalness results beat Moshi [7]; SALM-Duplex achieves duplex behaviour *without* special tokens, by treating the two streams as a single sequence and letting the model learn turn structure [15]; SALMONN-omni's codec-free variant reports the largest relative gains [3]. DuplexMamba uses explicit interrupt-vs-ignore state tokens and reports the best interrupt/ignore F1 among open duplex models [57]. The field has not converged on whether explicit control tokens help or hurt — both camps produce state-of-the-art claims.

**Single-stage recipes are viable.** VoiceTextBlender (NAACL 2025) shows single-stage joint speech-text SFT can beat larger multi-stage speech LMs [39]; SLAM-Omni trains a voice interaction system in a single stage with timbre control [41]; F-Actor's single-stage SFT recipe [49] and Slamming's 24-hour recipe [38] both argue the multi-stage pipeline is not sacred. This matters because the dominant industrial recipe (Moshi-style pretrain → SFT → alignment) is expensive; the evidence says a competent duplex model can be produced from an existing text LLM with far less.

## Theme 5 — Turn-taking, interruption, and alignment

This is where the field's centre of gravity now sits (for a dedicated review of turn-taking models see [60]). Three sub-themes.

**Engineered turn-taking components.** A large body of work builds specialised predictors rather than training them into the LLM: VAP-style voice-activity projection fine-tuned for backchannels (Yeah-Un-Oh, NAACL 2025 — F1 38.1 for continuers on real Japanese listening data) [40]; VAP extended to triadic multi-party conversation [19]; prompt-guided turn-taking models that work on LLMs without fine-tuning but are fragile to noise (84% → 52% accuracy at 10 dB music) [20]; neural-codec-based streaming end-pointers with label-delayed training that cut cutoff errors ~40% and shaved 1.2 s off a speech LLM's median response time [23]; Easy Turn (ICASSP 2026), an open 1,145-hour turn-state corpus plus a bimodal acoustic+linguistic detector reaching ~96% per-state accuracy [25]; JAL-Turn, a joint acoustic-linguistic detector that beats LLM-based detectors at 1/5 the latency [102]; and Semantic-Aware Interruption Detection, which introduces an interruption benchmark and an "average penalty time" metric on which an LLM-based detector beats Freeze-Omni's VAD by ~3× [103]. LLM-Enhanced Dialogue Management (2025) shows a 0.5B semantic-VAD dialogue manager can distinguish real from fake barge-ins [85]. The engineering consensus: acoustic-only VAD is too dumb for duplex; adding linguistic/semantic signal at low latency is necessary, and cheap models suffice.

**Learned turn-taking via post-training alignment.** The 2026 wave applies RL to interaction behaviour. Multi-Faceted Interactivity Alignment uses GRPO over four axes (pause, turn-taking, backchannels, interruption) with axis-specific timing rewards plus an LLM-judge semantic reward; it improves every interactivity metric on both Moshi and PersonaPlex, and the ablation shows the semantic reward is load-bearing — timing-only rewards degrade content [76]. ASPIRin projects the RL action space to binary speak/silence, avoiding the generative collapse that raw-token GRPO causes [81]. DuplexPO (Decoupling Conversational Dynamics) uses an RL objective that decouples turn dynamics from content, reporting 0.24 s latency on Full-Duplex-Bench v3 while *improving* reasoning [80]. Aligning Spoken Dialogue Models from User Interactions (ICML 2025) applies offline preference optimisation (DPO/SimPO) to preference pairs harvested from live Moshi interactions, gaining on both QA and safety [8]. Dual-Axis Generative Reward (ACL 2026) trains a reward model on both semantic and turn-taking axes with CoT + GRPO [50]. BayLing-Duplex's DPO stage [71], MiniCPM-o 4.5's GRPO [70], and SALMONN-omni's DPO [3] all report alignment as the step that makes duplex behaviour *reliable* rather than emergent. Behavior-SD (NAACL 2025) and its successor line (S-MARC [74], F-Actor [49], Enabling Conversational Behavior Reasoning [82]) contribute behaviour-annotated data and controllable behaviour targets for these objectives. An analysis of two connected Moshi instances shows their internal representations synchronise near zero lag and that anticipatory turn-taking signals are decodable from delayed activations — evidence that learned turn-taking readiness exists inside these models [78].

**What the benchmarks show about interruption behaviour.** The consistent finding across Full-Duplex-Bench [22], FLEXI [98], FD-Bench [17], the HumDial study [101], and MTR-DuplexBench [48] is that open-source duplex models interrupt mostly during pauses, lose semantic coherence when interrupted mid-stream, and rarely produce well-timed backchannels — while commercial systems (Gemini Live, GPT-4o-class) stop promptly on barge-in. Talking Turns (ICLR 2025) found Moshi interrupts too aggressively and mostly *keeps speaking* after user interruptions [10]. EchoChain (2026) shows no open model exceeds ~50% pass rate on state-update reasoning under mid-speech interruption [109]. The "chronological thinking" line of work targets exactly this failure — keeping dialogue state coherent across interruptions — by generating thinking chains during the listening window [83] [84] [77]. IRAF addresses the acoustic side, with interference-resilient fusion that improves duplex response quality under noise [110].

## Theme 6 — Latency: the number everyone quotes and nobody standardises

Reported latencies in this corpus range from ~200 ms (Moshi theoretical [1], NTPP [7], SoulX-Duplug [73]) to ~800 ms practical (MinMo [66]) to multiple seconds (VITA-class models in third-party benchmarks [98]). The components of that number are understood: chunk size (Moshi 160 ms [1], Freeze-Omni 160–320 ms [63]), codec rate (12.5 Hz → 80 ms per token [1] [69]), decoding parallelism (multi-token prediction [44], hidden-state reuse in Thinker-Talker [64] [65]), and the end-pointer's decision latency (Easy Turn 263 ms [25], JAL-Turn 36–38 ms [102], SoulX 240 ms [73]). X-Talk (2025) makes the modular case quantitatively: an event-driven cascade with no training at all achieves 284–610 ms end-to-end [88]. The survey's real-time rule of thumb — under 200 ms of compute per output token — is a useful bound [104].

But the numbers are not comparable, and the benchmark family keeps growing: Full-Duplex-Bench v2 adds a multi-turn automated-examiner protocol [54]; MTR-DuplexBench [48] and M3-DuplexBench [100] segment continuous dialogue into turns; URO-Bench evaluates end-to-end spoken dialogue understanding and generation [45]; VoiceBench benchmarks voice assistants under accents and noise [56]; Audio MultiChallenge (ACL 2026) evaluates multi-turn natural interaction [53]. Full-Duplex-Bench measures response latency to scripted interruptions [22]; FLEXI measures takeover latency in six scripted scenarios [98]; FD-Bench measures interruption-response delays on synthetic users [17]; MTR and M3 measure turn-taking latency by turn [48] [100]; τ-Voice measures task completion under a simulated user [99]; the HumDial study measures stop latency on real recordings [101]. Each defines its own clock. This is the clearest gap in the field: until a benchmark convention is adopted, "latency" claims across papers are not evidence about the same quantity.

## Where the evidence disagrees

**End-to-end vs cascaded.** X-Talk argues modular S2S can match E2E with far less data [88]; SOVA-Bench and RealTalk-CN find cascades retain more knowledge/content ability [18] [52]; the HumDial study's winning systems were largely modular [101] [26]; and SoulX-Duplug [73] and UAF [72] show module-based duplex control beating E2E on turn metrics. Against this, the E2E camp points to naturalness (NTPP [7]), the latency floor of cascades (X-Talk's own 284–610 ms vs ~200 ms E2E [88]), and the coherence failures of E2E under interruption [22] [109]. The apparent conflict largely dissolves once you separate *content quality* (cascades win, because the text LLM is stronger) from *interaction quality* (E2E wins on latency and naturalness, cascades win on interruption robustness). The 2026 "semi-cascaded" systems (Unit-Based Agent [27], SoulX [73]) are explicitly attempts to take both.

**Learned vs engineered turn-taking.** FLEXI found Gemini's VAD-based strategy stopped on interruption far more reliably than open models with learned turn-taking [98]; Full-Duplex-Bench v1.5 frames the same split as "responsive vs floor-holding" strategies [30]; yet the alignment papers show learned turn-taking improving under RL [76] [80], and Semantic-Aware Interruption shows learned semantic detection beating VAD by 3× on its own metric [103]. The disagreement is partly metric-driven (robustness vs responsiveness) and partly era-driven: 2026 learned systems are better than 2024 learned systems, and the benchmarks were built on the 2024 ones.

**Synthetic vs real data.** Covered in Theme 3; the camps are real-data purists [105] [87] vs scale-through-synthesis [95] [93] [49], with DuplexChat [96] as the compromise (real audio, automatically separated).

**Control tokens vs none.** DuplexMamba [57] and the state-token systems (UAF [72], BayLing-Duplex [71]) report explicit tokens help; SALM-Duplex reports no-token duplex works better [15]; NTPP's joint objective is a third position [7]. No independent replication exists to adjudicate.

**What the disagreement itself shows.** The field has not converged on a shared experimental protocol: different benchmarks, different baseline versions of Moshi, different synthetic-data pipelines, and self-reported latency conventions. Statements like "our model beats Moshi" are usually true of a specific benchmark configuration, not of the model generally.

## Gaps and open questions

1. **No standardised duplex evaluation.** Thirteen benchmarks, thirteen clocks (Theme 6). What would settle it: a shared protocol with a common interruption corpus, common latency definition, and common model-as-service harness — the HumDial challenge [29] is the closest attempt.
2. **The real-data bottleneck is unresolved.** Only a handful of resources contain genuine human overlap (Fisher, DuplexChat's automatically separated podcasts [96], HumDial recordings [101], Human-1 [87]); whether separation artefacts corrupt duplex learning is untested.
3. **Content-vs-timing coupling.** Alignment that improves turn-taking can degrade knowledge and safety [76] [48]; the mechanism (what in the RL objective causes the trade-off) is not established.
4. **Long-horizon degradation.** MTR shows multi-round performance collapsing across rounds [48]; no paper yet explains or fixes state drift in continuous duplex sessions.
5. **Multi-party duplex** exists only as perception (triadic VAP [19]); no open model does three-way overlap.
6. **Independent replication.** Almost every headline number (latency, naturalness, TOR) comes from a single paper, often the proposing lab. The one replicated baseline — Moshi — is now two years old.
7. **Unretrievable frontier work.** Several systems discussed in surveys and on social media (e.g., Hume's "FullDuplex") are absent from all indexed databases; the reviewable record and the visible frontier are not the same set.

## Confidence and limitations

Confidence in the three headline findings is moderate-to-good: the data-constraint claim is made independently by two surveys and consistent with the training-data ablations in multiple system papers [105] [104] [95] [96]; the engineered-vs-learned split is visible across at least eight independent systems; the evaluation incoherence is directly observable in the benchmark definitions. Confidence in specific numbers is lower: 32 of 110 sources were read at abstract level only (paywalled IEEE/ACM/MIT-Press items and four ACL/EMNLP papers whose full text could not be retrieved), and where the review cites their claims it does so hedged. Several quantitative claims rest on single papers with self-defined metrics and no external replication; the review treats those as reported results, not established facts. Venue accuracy was verified per-record, but two NeurIPS 2025 DOIs (VITA-1.5, SALMONN-omni) resolved through Crossref's proceedings records rather than OpenAlex, and their page numbers could not be cross-checked. The unretrievable-anchor problem cuts both ways: this review may under-represent work that exists only on project pages or in non-indexed venues. Finally, the field moves fast enough that a review frozen on 2026-08-07 will be dated within quarters; the chronological framing above is meant to make that visible rather than to hide it.

## Evidence table

| key | year | design | sample | measure | finding | limitations | confidence |
| --- | --- | --- | --- | --- | --- | --- | --- |
| aec2025echo | 2025 | signal-processing model (small-footprint streaming AEC for mobile full-duplex) | mobile full-duplex AEC task with nonlinear distortion and reference latency; small-footprint streaming RES model + progressive learning + Wiener post-filter | PESQ, ERLE, VAD DCF, ASR WER | Progressive learning (stages +10/+20/+∞ dB) reduces ASR WER to 8.83% (vol70) and 15.81% (vol100), and task-specific post-filter parameters (β=0.6 VAD, 0.2 ASR) further improve downstream VAD/ASR, with ERLE up to 45.93 dB. | mobile-specific; optimal post-filter β differs per downstream task (PESQ-optimal 0.4 ≠ WER-optimal 0.2); paper reports its own testbed | high |
| aspir2026 | 2026 | ASPIRin: interactivity-optimized RL — Action Space Projection maps text vocabulary to a binary active-speech/silence state policy optimized with GRPO and rule-based rewards (interruption overlap and response latency, tau=1.0 s); LoRA on Moshi | 43 h in-house dual-channel conversational data (~1,300 two-minute clips); Moshi base; 8 V100, 3 epochs, GRPO group G=2 | Full-Duplex-Bench TOR and latency across pause/backchannel/turn-taking/interruption; GPT-4o 1-5 semantic rating; duplicate n-gram ratio (seq-rep-n) and Self-BLEU | ASPIRin reduces duplicate n-grams by >50% vs standard GRPO (eliminating repetitive collapse), cuts interruption latency by ~0.2 s, and balances TOR (0.273 vs 0.153 GRPO on turn-taking; 0.364 vs 0.709 on backchannel) with only a 0.16 GPT-4o score drop. | Tiny 43 h dataset; binary speak/silence abstraction discards fine-grained control; single base model (Moshi); no reasoning/QA evaluation of the aligned model | high |
| audiopalm2023 | 2023 | fusion of PaLM-2 text LLM and AudioLM: single decoder-only transformer over a joint text+audio token vocabulary (w2v-BERT/USM semantic tokens at 25Hz, 1024 vocab) with task markup; weights initialized from text-only PaLM-2 | 8B model trained on mixtures of ASR/AST/TTS/S2ST corpora (VoxPopuli, CVSS, CoVoST2, CommonVoice, YouTube ASR, WMT/TED, Conversational EsEn); exact hours not stated in text read | BLEU (CoVoST2/FLEURS AST), WER (ASR), speech quality and voice similarity (objective + human MOS) | Initializing from text-only PaLM-2 weights improves speech processing, yielding SOTA results on AST and S2ST benchmarks with zero-shot speech-to-text translation on unseen language pairs and cross-lingual voice transfer, significantly outperforming Whisper on AST-observed FLEURS languages. | Turn-based task prompting; no streaming or full-duplex interaction; tokenization choice (w2v-BERT vs USM) strongly affects performance; not designed for dialogue. | high |
| baichuan2025 | 2025 | Omni-modal MLLM; LLM decoder alternately predicts text and audio tokens; 8-layer RVQ Baichuan-Audio-Tokenizer at 12.5 Hz; multi-stage pretrain+SFT | 7B LLM; ~500B multimodal tokens; audio pretraining 887K hours (ASR 185K, audio-text interleaved 393K, TTS/ITTS 193K) | Average scores across text/image/video/audio/medical benchmarks; OpenAudio-Bench | Baichuan-Omni-1.5 scores avg 73.3 on ten image-understanding benchmarks (surpassing GPT-4o-mini by ~6 pts) and 83.8% on OpenMM-Medical with a 7B LLM (vs Qwen2-VL-72B's 80.7%), supporting controllable bilingual real-time voice interaction. | No explicit full-duplex metrics (turn-taking, interruption latency) in the paper; full-duplex claims are qualitative; focus is modality parity and medical benchmarks. | high |
| bayling2026 | 2026 | Native full-duplex single autoregressive LLM; multi-channel interleaved sequence (user-speech:assistant-text:assistant-speech blocks, N=10/M=5) with only 4 added dialogue-state tokens; SFT then DPO | GLM-4-Voice backbone (9B); 400K full-duplex samples for SFT + lightweight DPO stage | Turn-taking success rate, interruption success rate, speech-response score, spoken QA accuracy | BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, lifts the speech-response score from 2.17 (Moshi) to 3.39, and reaches 46.0%/18.1% on Llama/Web Questions vs Moshi's 21.0%/9.2%, while matching its turn-based counterpart on standard benchmarks. | Conversion recipe assumes a strong turn-based SpeechLM (GLM-4-Voice) checkpoint; 400K synthetic samples may not capture acoustic diversity of real overlapping speech. | high |
| behavior2025 | 2025 | Graph-of-Thoughts conversational-behavior reasoning framework: hierarchical speech-act detection (HuBERT+Whisper features, 1-s chunks, causal Transformer, high-level intents + low-level acts) plus causal GoT reasoning that generates interpretable rationales | 28,000 synthetic clips / 192 h (GPT-4o transcripts + CosyVoice2, event-rich: 17.6% interruption) with 37,100 human-validated rationale entries + 118 h real CANDOR subset | Per-class F1 / ROC-AUC (high & low-level acts), BLEU/ROUGE/cosine similarity of generated rationales, human ratings of rationales | Low-level speech-act detection reaches macro-F1 0.660 / micro-F1 0.768 / interruption F1 0.515 on synthetic data and AUC 0.796 (turn-taking), 0.701 (backchannel) on real CANDOR data, with human-validated rationale chains. | A perception+reasoning module, not a generative full-duplex dialogue model; interruption detection F1 only 0.515; heavy reliance on synthetic data and manual rationale annotation | high |
| chen2025aligning | 2025 | Offline preference alignment (DPO / SimPO / APO-Zero) of a Moshi full-duplex model using AI-feedback preference pairs spanning both linguistic content and temporal dynamics (interruption, unresponsiveness) | 283,740 preference pairs total (154,301 unique-context; final mix 93,490) mined from live user interactions with deployed Moshi; Moshi 7B temporal + 600M depth transformers; 8 h of human-eval dialogues | Spoken QA accuracy (Llama Questions, TriviaQA, WebQuestions), safety (ALERT, XSTest via WildGuard), human ratings of coherence & flow, interruption behavior | Preference alignment improves QA by avg 3.1% across 3 benchmarks and safety by avg 6.9% across 2 benchmarks, with human evaluation confirming better real-time interaction behavior (e.g., fewer talk-overs) beyond single turns. | Preferences come from an LLM judge (Mistral Large 2) rather than human labels; user audio is discarded and resynthesized via ASR+TTS (noise); 57% of timing-only pairs rely on programmatic detection | high |
| chrono2025 | 2025 | CT-Duplex: full-duplex SDLM (Qwen2.5-1.5B LLM backbone + streaming encoder at 12.5 Hz + autoregressive Transformer speech decoder, Nanocodec) where ACT-R-inspired 5-node-type thinking chains replace silence tokens during the listening phase (causal, amortized, no added latency) | Synthetic training data GenConv 10.5k h + SpokenWOZ-G 2.0k h + Llamaq-G 2.7k h (Qwen2.5-72B generated, Step-Audio-3B TTS, 50K+ speaker library); 1.7B total params; 8 L40s | GPT score / BLEU / Sentence-BERT on SpokenWOZ & MtBenchEval, accuracy on Llama/Web Questions, turn-taking & barge-in latency and success rate (Impatient set), A/B human test | Chronological thinking raises SpokenWOZ GPT score 2.40->2.61 (+8.75%) and Llama Questions accuracy to 31.4% (vs SALM-Duplex 15.0%, Moshi 21.0%) while keeping barge-in success 94.05% with no extra latency (turn-taking 0.68 s, barge-in 0.54 s). | Entirely synthetic training data; 1.5B backbone caps absolute scores; negligible gains on factual QA; thinking chains pre-generated by a 72B teacher | high |
| cotfd2025 | 2025 | SCoT: blockwise (2 s) time-multiplexed duplex E2E SDS with streaming Chain-of-Thought — CTC-forced-alignment-derived intermediate targets (ASR transcript and/or text response) per block; SmolLM2-1.7B init, delay-interleave multi-stream, ESPnet-Codec + XEUS SSL tokens | Real human-human data: Switchboard (~300 h) + Fisher (~2,000 h), combined train; 1.7B model; 4 H200 GPUs | ROUGE-1/2/L, METEOR, perplexity, Qwen2.5-7B-Omni judge score, RTF (latency), overlap %/precision/recall, UTMOS, emotion rank | SCoT-Response raises ROUGE-1 to 27.9 and METEOR to 26.4 (duplex E2E baseline: 14.4/11.8) at RTF 0.56 (below turn-by-turn E2E's 0.75), while SCoT-Full best matches natural overlap (64.5% blocks, recall 71.5 vs 57.5% in Eval2000) at RTF 0.85. | Higher RTF than plain duplex E2E (0.29); perplexity inflated by spoken fillers/backchannels; 1.7B scale; no RL or alignment stage; ASR-only CoT variant degrades quality | high |
| decouple2026 | 2026 | DuplexPO: RL framework decoupling when-to-speak from what-to-say — GRPO-style policy optimization with Factorized Conversational Dynamics Reward (FCDR) over dynamics-critical windows; pipeline = pretraining, SFT, DuplexPO RL, speech-synthesis SFT (Qwen2.5-7B + 600M Parakeet streaming encoder + CosyVoice2) | 530K h speech-continuation + 70K h synthetic QA pretraining/SFT; RL on 24.6K Fisher + 43.1K Seamless windows (human conversations); 64 A800 GPUs | Window-level onset MAE / init rate / yield rate (Fisher, Seamless), FDB-v3 turn-taking latency / Voiced Interrupt Rate / yield, Gemini-3.0 pairwise judge, QA + instruction-following + reasoning benchmarks | DuplexPO achieves 100% turn-take and 100% yield at 0.24 s FDB-v3 latency (vs GPT-Realtime 2.65 s, Moshi 0.44 s, SFT baseline 7.33 s) while slightly improving intelligence (LlamaQ 75.3 vs 72.0, OBQA 73.7 vs 72.2 over SFT baseline). | Most training data synthetic (LLM-generated + TTS); dynamics evaluation is window-based and partly self-referential; ablations show DPO objective and learned reward models weaker than FCDR+GRPO | high |
| dgslm2023 | 2023 | textless dual-tower transformer with cross-attention over two speaker channels; discrete units from unsupervised spoken unit discovery (k-means over HuBERT); generates both channels simultaneously without text | 2000h two-channel Fisher conversational audio, no text/labels; model size not reported in abstract | naturalness/fluidity of turn-taking and paralinguistic signals vs text-based cascaded model | First textless model to generate naturalistic two-channel spoken dialogue: trained on 2000h of Fisher two-channel audio, it produces speech, laughter, and paralinguistic signals in both channels simultaneously with more naturalistic turn-taking than a text-based cascade (abstract claim). | Abstract notes only comparative naturalness; later local works (pseudostereo2024, sync2024llms) report dGSLM struggles with semantic coherence due to limited data/units, and SyncLLM found its turn-taking correlation low on out-of-distribution data. | moderate |
| dualaxis2026 | 2026 | Generative dual-axis reward model for full-duplex spoken dialogue models: outputs two CoTs (semantic coherence + turn management) and a binary score S; trained on Qwen-2.5-Omni-7B via 3 stages - SFT-1 interaction grounding (event detection/diarization), SFT-2 CoT distillation (2,670 samples), GRPO RL | Training data: 6,361 synthetic samples (~146h, programmatically timestamped) + 100 human-human (Seamless Interaction) + 289 human-machine samples (~10h, manually annotated) | Accuracy and macro-F1 on ID/FG/OOD/RW-HH/RW-HM test sets; CoT consistency score and win rate vs human experts | Full model reaches 98.54% accuracy in-distribution, 86.79% on real human-human and 77.27% on real human-machine dialogues (vs 7.33%/49.06%/52.27% for the base Qwen-2.5-Omni), with GRPO ablation dropping ID accuracy by 21.90 points. | Majority synthetic training data; binary score is coarse; reward model validated as a judge - end-to-end RL training of an SDM with this reward is not demonstrated; only ~10h of real human-machine data. | high |
| duplex2024models | 2024 | Duplex model adaptation of LLMs via time-division multiplexing (TDM): queries and responses divided into time slices processed pseudo-simultaneously; fine-tuning dataset of alternating query/response slices covering typical instantaneous feedback types | Fine-tuning dataset of sliced conversations (scale not stated); LLM size not stated | Preservation of standard benchmark performance + automatic and human evaluation of interaction naturalness and user satisfaction (no numbers in abstract) | With only a few fine-tuning steps LLMs retain standard-benchmark performance while duplex models markedly improve naturalness and user satisfaction over vanilla LLMs (no numbers in abstract). | No metrics in abstract; pseudo-simultaneous time-slicing rather than true acoustic overlap; text-level duplex without speech codec integration described. | moderate |
| duplexchat2026 | 2026 | data pipeline + corpus (speaker-separated FD dialogue speech) | DuplexChat corpus: 282,634h English + 132,723h Japanese speaker-separated two-speaker dialogue (~415k h total) from podcast feeds; diarization-guided extraction + separation/restoration | corpus hours, channel separation quality, turn-taking dynamics analysis | DuplexChat-Pipe yields the largest open dialogue resource to date (~415k h, vs Fisher 2k h and Spotify Podcast 60k h), with analysis showing it preserves human-like turn-taking dynamics. | podcast-domain audio (not spontaneous dyadic phone speech); separation/restoration may introduce artifacts not quantified in available text | high |
| duplexmamba2025 | 2025 | Mamba-based end-to-end duplex model (ConMamba speech encoder + Mamba LM, 2.8B) with duplex decoding via input state tokens; four-stage training: ASR, voice assistant, input state discrimination (interrupt vs ignore), streaming alignment | ~11k hours ASR data (LibriSpeech, TED-LIUM 3, Multilingual LibriSpeech) + VoiceAssistant dataset + GigaSpeech-L ignored-data; 2.8B parameters | VoiceBench (GPT-scored QA, SD-QA accuracy, IFEval), LibriSpeech WER (test-clean/test-other), interrupt/ignore precision-recall-F1, GPU memory vs context length | DuplexMamba ranks 3rd overall on VoiceBench behind DiVA and Qwen2-Audio (8B/7B Transformers) and best among duplex models, with interrupt/ignore F1 of 99.35% vs 90.47% (VITA) and 79.32% (Qwen2-Audio). | Weak SD-QA due to limited internal knowledge in fine-tuning data; duplex stages degrade ASR (stage-4 test-clean WER 4.94 vs 3.36 at stage 2); speech-to-text output only. | high |
| easy2026turn | 2026 | turn-taking detection model + dataset release | Easy Turn trainset 1,145h (complete 580h/423k, incomplete 532h/712k, backchannel 10h/41k, wait 23h/40k); testset 800 (300/300/100/100); Whisper-Medium + Qwen2.5-0.5B, 850MB model | per-state accuracy (complete/incomplete/backchannel/wait), parameters, latency (263ms), memory | Easy Turn reaches 96.33/97.67/91/98 state accuracies vs TEN Turn Detection (86.67/89.3/-/91) and Smart Turn V2 (78.67/62/-/-), and the ASR+turn-detection paradigm adds ~8 points over state-only prediction (95.75 vs 87.88 avg). | testset is self-released; labels generated via LLM cross-annotation which can inherit annotation-model biases; Mandarin-centric data | high |
| echochain2026 | 2026 | Benchmark (controlled evaluation harness) for full-duplex state-update reasoning under mid-speech interruptions: planner-simulator loop generates scenario-driven conversations, barge-in injected at fixed offset from assistant speech onset, GPT-5.2 failure classifier + human rubric review; failure taxonomy (contextual inertia, interruption amnesia, objective displacement) | 200 interrupted conversations; 4 models evaluated (GPT-realtime-2025-08-28, Gemini Live-2.5-flash-native-audio, Grok Voice Agent, Amazon Nova Sonic 2); paired half-duplex control on 48 conversations | Mean pass rate (MPR), mean criteria pass (MCP), failure counts per taxonomy category | No model exceeds 50% MPR (best: Grok Voice Agent at 48.5%), and the paired control shows interrupting identical content causes 40.22% more failures than half-duplex delivery (92 vs 55 failures across 48 conversations). | Evaluates only task-informative interruptions at a fixed barge-in offset; excludes backchanneling, side-speech, and ambient noise; relies on LLM (GPT-5.2) planner and classifier with human review only for flagged cases. | high |
| endpointer2025 | 2025 | turn-taking model (streaming endpointer on neural audio codec features) | multi-turn dialogue test sets; single-stream and two-stream endpointer configurations; integration with codec-based speech LLM | median latency (fixed 160ms), cutoff error rates, response time | At a fixed 160 ms median latency, neural-codec features plus label-delayed training cut cutoff errors by 42.7% (single-stream) and 37.5% (two-stream), and integration improves a codec-based SLM's median response time by 1200 ms and cutoff error by 35%. | abstract-only; numbers from abstract only; evaluation sets not fully specified locally | moderate |
| factor2026actor | 2026 | First open instruction-following full-duplex conversational model: Llama3.2-1B-Instruct backbone + frozen audio encoder, dual user/system NanoCodec DAU streams with per-stream embeddings, 8 linear heads, optional text stream; single-stage SFT only (LLM + heads trained) | Behavior-SD dataset 2,164h (74.4% kept after forced-alignment filtering, ~2,000h), 4xA100-40GB, ~48h training per model | DAU/text perplexity, UTMOS, WER(audio vs text), speaking-time balance, dialogue-initiation accuracy, BC/interruption count correlation (Pearson r), narrative adherence (LLM judge + human eval on 80 dialogues) | Best configuration (word-level alignment, audio delay 2, s/u loss) reaches text PPL 21.45, WER 7.59%, >99% initiation-instruction accuracy, and backchannel-frequency correlation 0.54 (interruptions only 0.25, averaging 0.9 per conversation in training data). | Synthetic TTS-only training data (CosyVoice); interruptions too rare in data for strong control; 1B backbone; BC/interruption event detection is fragile (VAD threshold grid-search needed). | high |
| fdb152026 | 2026 | benchmark (overlap handling) | 5 state-of-the-art voice agents (named in paper, not in abstract) | categorical dialogue-behavior metrics, stop/response latency, prosodic adaptation | Benchmarking five agents across 4 overlap scenarios (user interruption, user backchannel, talking-to-others, background speech) reveals two divergent strategies: a responsive approach prioritizing rapid response and a floor-holding approach that filters overlapping events. | abstract-only; model names and numeric scores not available in local text | moderate |
| fdb2025bench | 2025 | benchmark (turn-taking evaluation) | 4 SDMs (dGSLM trained on 2,000h phone calls; Moshi 7B; Freeze-Omni; Gemini Live); data: Candor 850h, ICC 28.33min/118 speakers, 200 synthetic interruption + 137 pause samples | TOR (takeover rate), backchannel Freq, JSD, response latency, GPT-4o contextual relevance | dGSLM and Moshi interrupt most during pauses (TOR 0.934/0.985) with ~0.3s response latency while Freeze-Omni's explicit turn-control module lowers TOR, and end-to-end models lose semantic coherence under user interruption (GPT-4o scores: Moshi 0.765, dGSLM 0.201). | metrics are descriptive, not linked to human preference; English-only; small synthetic interruption set | high |
| fdb22026 | 2026 | benchmark (multi-turn with automated examiner) | not stated in available text (ACL 2026 short paper; only citation metadata locally) | automated examiner-based interaction; conversational features + instruction following + safety (per citing papers) | Proposes a multi-turn evaluation framework for duplex dialogue systems that uses an automated examiner to drive dynamic interactions; concrete scale and scores not available in local text. | only citation metadata available locally; cannot verify methods or numbers | low |
| fdbench2025pipeline | 2025 | benchmark pipeline (LLM+TTS+ASR automated) | 293 simulated conversations, 1,196 interruptions (5 types), ~40h synthetic speech, 3 SNR noise conditions; models Moshi (80ms chunks), Freeze-Omni (107ms, VAD 0.8), VITA-1.5 (200ms, VAD 0.7) | SRR/SRIR/SIR/EIR/NIR rates, IRD/FSED/ERT/EIT delays, WER, conditioned PPL, GPT-4o score | Moshi's success-reply rate falls from 61.7% (easy) to 34.1% (hard interruptions) and all three open-source FDSDS struggle under frequent disruption and noise (GPT-4o scores: Moshi ~4.4, Freeze-Omni ~3.3, VITA-1.5 ~2.4). | user side is entirely TTS-synthesized; 3 models only; metrics need timestamp post-processing | high |
| fdsurvey2026 | 2026 | survey (full-duplex SDS architecture, ontology, decision space) | audit across 15+ claimed full-duplex systems and FD benchmarks; L0-L3 architectural hierarchy; T x I x R interaction ontology; IDLE/LISTEN/SPEAK/WAIT/DUAL state machine | qualitative audit; realization gap between architectural capability and observed behavior; training-data coverage | The audit documents a realization gap: many architectures can in principle operate full-duplex, but observed behavior is constrained by the interaction patterns in training/evaluation data, with public training data far smaller than undisclosed industrial corpora. | qualitative audit, no quantitative benchmark scores; industrial corpora undisclosed by definition | high |
| flexduo2025 | 2025 | FlexDuo: pluggable full-duplex control module (Qwen2-audio-7B, frozen audio encoder) with 3 states (Speak/Idle/Listen) + 7-action FSM, 120 ms decision interval, drives any half-duplex LLM; explicit Idle state filters backchannels/noise | Fisher EN 671 h (4.79M state labels) + Fisher ZH 263 h (3.26M labels), GPT-4o-filtered (84% human acceptance); Qwen2-audio-7B; 8xH800, 40K steps | Turn-taking Positive-F1@1/5/10 (user/assistant), false interruption rates, combined scores, dialogue-quality conditional perplexity | FlexDuo reaches combined turn-taking F1 0.79 and combined false-interruption rate 0.30 (vs 0.32/0.53 for VAD-controlled half-duplex and 0.71/0.61 Freeze-Omni), corresponding to the paper's reported 24.9% false-interruption reduction and 7.6% response-accuracy gain. | Needs stereo-annotated training data (VAD + GPT-4o labeling pipeline); dialogue content still comes from an external half-duplex LLM; evaluated on Fisher only; Idle-state threshold tuning (500 ms) is heuristic | high |
| flexi2025 | 2025 | benchmark (6 full-duplex human-LLM scenarios incl. emergency model-interrupt) | synthetic user queries (Qwen-plus + CosyVoice2); models Moshi, Freeze-Omni, VITA-1.5, Gemini (commercial) | TOR/TTR, latency, GPT-4o Topic Shift Score (TSS) and Emergency Detection Score (EDS), backchannel JSD | Gemini terminates on user interrupt far better (TTR 0.935 vs Moshi 0.462, Freeze-Omni 0.539, VITA-1.5 0.226) and open-source models rarely produce aligned backchannels (JSD 0.947-0.986), with VITA latency ~4s vs ~1s commercial. | fully synthetic data; only one commercial model due to budget; backchannel timing judged by GPT-4o annotations | high |
| freezeomni2024 | 2024 | frozen Qwen2-7B-Instruct backbone; chunk-wise streaming speech encoder (350M, 12.5Hz output) + adapter; VALL-E-style speech decoder (NAR prefix + NAR + AR, 120M) with single-codebook TiCodec (40Hz, 24kHz); three-stage training for speech-in and speech-out; duplex via chunk-level state prediction (VAD-triggered, states 0/1/2) with 'model-as-a-server' deployment | 110,000h ASR speech-text paired data (stage 1-2 speech input), ~3,000h TTS text-speech paired data, only 60,000 multi-round text Q&A items (moss-003-sft) synthesized to speech; 8 GPUs | ASR CER/WER on AISHELL-1/test_net/test_meeting and LibriSpeech dev/test; CER of synthesized speech; spoken QA accuracy; end-to-end latency breakdown (ms) | Fully frozen-LLM speech-to-speech dialogue: total end-to-end response latency 745ms average (753 median, 1020 90th pct; LLM-interrupted-to-first-text-chunk 478ms, first speech token chunk 237ms), with spoken QA accuracy (44.73 Web Q / 72 Llama Q / 53.88 Audio Trivia) close to its text-only backbone, and duplex handled by chunk-level user-interruption state prediction. | Single-speaker speech output only; state prediction depends on an external VAD; duplex is a decision module rather than true simultaneous dual-stream generation; latency higher than Moshi/LLaMA-Omni. | high |
| funaudio2024 | 2024 | model family for voice interaction: SenseVoice (non-autoregressive encoder-only ASR+LID+SER+AED; SenseVoice-Small / SenseVoice-Large), S3 semantic speech tokenizer, CosyVoice (300M flow/LLM-based TTS with zero-shot in-context learning and instruction control); components designed to be glued around an LLM | SenseVoice trained on 300k+ hours; CosyVoice trained on 170k+ hours across 5 languages (ZH/EN/JA/Yue/KO); CosyVoice-base-300M and instruct-300M open models | multilingual ASR accuracy, emotion recognition, audio event detection, TTS naturalness/emotion controllability | SenseVoice-Small delivers low-latency multilingual ASR (5 languages) and SenseVoice-Large high-precision ASR (50+ languages) from 300k+ hours, while CosyVoice (170k+ hours, 300M params) enables 3-second-prompt zero-shot voice cloning and emotion/style control - the open building blocks of FunAudioLLM's cascaded voice applications. | Components are not a single end-to-end model; duplex behavior and low latency depend on system integration rather than the models themselves. | high |
| glmvoice2024 | 2024 | GLM-4-Voice: end-to-end spoken chatbot from GLM-4-9B text LLM; ultra-low-bitrate 175bps single-codebook speech tokenizer at 12.5Hz built by inserting a VQ bottleneck into an ASR encoder; speech-text interleaved data synthesized from text corpora via text-to-token model; continued pretraining + conversational SFT | GLM-4-9B; continued pretraining scaled to 1 trillion tokens (unsupervised speech + interleaved speech-text + supervised speech-text); open models released | speech language modeling perplexity, spoken QA, conversational ability and speech quality vs baselines | GLM-4-Voice reaches state-of-the-art speech language modeling and spoken QA by continuing pretraining GLM-4-9B on 1T tokens including synthesized speech-text interleaved data, using a 175bps single-codebook tokenizer that keeps audio token sequences short enough for LLM modeling. | Abstract reports no absolute latency/WER/MOS numbers; turn-based (no full-duplex claim); English+Chinese only; synthesized interleaved data quality depends on TTS. | moderate |
| gpt4o2024 | 2024 | autoregressive omni model trained end-to-end across text, vision, and audio (single network for all inputs/outputs); system card focusing on speech-to-speech safety evaluations | model size and training data not disclosed in system card | audio response latency (ms), text benchmark parity with GPT-4 Turbo, safety evaluations (voice generation, speaker ID, persuasion, etc.) | GPT-4o responds to audio inputs in as little as 232ms with an average of 320ms (similar to human conversational response time), matches GPT-4 Turbo on English text and code while being faster and 50% cheaper in the API, with improved non-English text and audio/vision understanding. | No architecture/training details disclosed; system card emphasizes safety risks (unauthorized voice generation mitigated by preset voices; reduced audio robustness to noise/echo; anthropomorphization). | high |
| holler2023turn | 2023 | empirical study of multimodal turn-taking in human face-to-face interaction (eye-trackers + multi-camera corpus; qualitative conversation analysis + quantitative measures) | corpus of multimodal face-to-face interactions; details in full paper (not in abstract) | transition speed and inhibition at points of possible completion (gaze direction, gesture onset/completion) | Turn transitions are inhibited when the speaker averts gaze or produces beginning/unfinished gestures at possible completion points, and turns with manual gestures have faster transitions - showing transition-relevance places are multimodal (linguistic + visual). | Observational human-interaction study; findings about humans inform, but do not directly measure, agent turn-taking systems. | high |
| hu2025codec | 2025 | X-Codec: acoustic codec (EnCodec/DAC-style) augmented with semantic features from a pretrained semantic encoder injected before RVQ plus a semantic reconstruction loss after RVQ; evaluated as drop-in tokenizer for audio LLMs (VALL-E TTS, music continuation, text-to-sound) | X-Codec model size/training hours not quantified in text read; evaluated with VALL-E zero-shot TTS on LibriSpeech test-clean | zero-shot TTS WER (ASR-transcribed), speaker similarity (Sim-O), UTMOS; phonetic discriminability ABX; music/sound generation quality | X-Codec's semantic-enhanced tokens significantly reduce WER in zero-shot TTS compared to EnCodec/DAC and SpeechTokenizer while improving Sim-O and UTMOS, and ABX phonetic discriminability improves as the number of quantizers increases - confirming acoustic codecs' semantic shortcoming hurts audio LLMs. | Semantic encoder adds inference/complexity; benefits shown mainly for TTS-style generation; VALL-E evaluation setting (continual) may not transfer to full-duplex streaming. | high |
| human12026 | 2026 | Human-1: Hindi adaptation of Moshi — custom 32k Hindi SentencePiece tokenizer, reinitialized text-vocabulary parameters, frozen Mimi codec, two-stage training (large-scale pretraining then fine-tuning) | 26,000 h real stereo spontaneous Hindi conversations from 14,695 speakers (pretrain, 1 epoch ~10K steps in ~13 h on 8xH100); 1,000 h curated fine-tune; 7B temporal transformer; 130 raters / 2,125 paired comparisons | PPL (Sarvam-1 2B on Whisper-v3 transcripts), human naturalness/clarity 5-point scores + conversational rubrics, turn-taking statistics (IPU/pause/gap/overlap), PESQ/STOI codec check | Human-1 scores naturalness 4.10 vs 4.55 for human speech (66.9% ties), 85% human-like-interaction pass rate, PPL 356.9 vs 237.1 ground truth at tau=0.8, with tau=0.9 turn-taking closest to ground truth (overlap 4.30 vs 3.03 s/min). | Text loss overfits after ~4,800 fine-tune steps; completion 42% and appropriateness 53% pass rates; 75% PAD-token ratio; moderate codec quality (PESQ 2.55) on Hindi | high |
| humdial2026 | 2026 | benchmark/challenge overview (ICASSP 2026 HumDial) | challenge dataset derived from authentic human conversations; two tracks (Emotional Intelligence; Full-Duplex Interaction) | long-term emotion understanding, empathetic generation, real-time listening-while-speaking decision metrics (details in companion study) | Launches the first Human-like Spoken Dialogue Systems Challenge benchmarking dual capabilities of emotional intelligence and full-duplex interaction under listening-while-speaking conditions; final results summarized in the paper. | abstract-only; no numeric results in available text | moderate |
| humdialstudy2026 | 2026 | benchmark + dataset release (HumDial-FDBench, Interruption/Rejection) | real dual-channel human recordings; 8 interruption/rejection scenarios with per-split instances (e.g., Follow-up 1507/200/600 train/dev/test); leaderboard incl. Gemini, Moshi, Freeze-Omni, 9 ranked teams | interruption-handling score, rejection-handling score, overall score, latency | On HumDial-FDBench Gemini leads interruption handling (79.8) and Freeze-Omni leads rejection handling (50.2) while Moshi scores low on both (35.4/22.8), and Gemini has the lowest latency (1.301s) via its VAD-based strategy. | performance degrades in multi-speaker environments and under transient noise; latency control under uncertain acoustics remains unsolved | high |
| icmi2023slow | 2023 | workshop paper (ICMI 2023) on deploying human-like conversational turn-taking in spoken agents; argues one-to-one speak/wait interaction causes long pauses and harms UX; reviews challenges and design guidance | n/a (position/workshop paper) | n/a (qualitative discussion of pause lengths and interaction flow) | Argues that standard one-at-a-time speak/wait conversational interaction with agents produces long between-turn pauses that undermine flow and naturalness, and calls for turn-taking-aware design (with LLM-era speech systems as the context). | Workshop position paper without controlled experiments or quantitative latency data. | low |
| inconsistent2025 | 2025 | Analysis and mitigation of Discrete Representation Inconsistency (DRI) in neural codec LMs: quantifies how one audio segment maps to multiple divergent token sequences (EnCodec), and proposes a mitigation method for codec-LM training | Codec LM trained on LibriTTS and large-scale MLS (44,000 hours) | Speech generation quality - omissions and repetitions (no numeric scores in abstract) | DRI causes omissions and repetitions in codec-LM speech generation, and the proposed mitigation effectively reduces it on LibriTTS and 44,000-hour MLS training (no numbers in abstract). | No metrics in abstract; single-stream generation focus rather than duplex overlap; analysis specific to EnCodec-style tokenizers. | moderate |
| interact2025corpus | 2025 | corpus (speech dialogue interaction) | not stated in available text (Findings of EMNLP 2025; only citation metadata locally) | not stated in available text | InteractSpeech is introduced as a speech dialogue interaction corpus for spoken dialogue models; scale and content details not available in local text. | only citation metadata available locally | low |
| interactalign2026 | 2026 | Post-training RL (GRPO) alignment across four interactivity axes — pause handling, turn-taking, backchanneling, user interruption — with axis-specific delay/divergence rewards plus an LLM-Judge reward for semantic quality; segments auto-extracted from human corpora | Applied to Moshi and PersonaPlex; training segments from Fisher and Seamless corpora; evaluated on Full-Duplex-Bench v1 (static) and v2 (real-time multi-turn with GPT-Realtime) | TOR, JSD vs ground truth, response latency, GPT-4o/Gemini-2.5-Flash semantic scores, UTMOSv2 | GRPO over the four axes improves every interactivity metric for both Moshi and PersonaPlex on FDB v1 and v2; ablations show removing the LLM-Judge reward causes the largest degradation across nearly all metrics, and Fisher training degraded PersonaPlex's safety-task instruction-following. | Fixed 4-axis scope; training on short segments needs preceding context to generalize to multi-turn; pure timing rewards can degrade semantics without an LLM-based quality reward. | high |
| iraf2026 | 2026 | End-to-end dual-channel full-duplex spoken dialogue model with Interference-Resilient Adaptive Fusion (IRAF): frame-level scalar reliability gate predicted from target-speaker and user audio embeddings rescales user audio before fusion with agent text embeddings; streaming-compatible, no added latency | TinyLlama 1.1B LLM + 100M streaming speech encoder + NanoCodec 0.6kbps; trained/evaluated on MS-MARCO (CosyVoice2-synthesized) and multi-turn InstructS2S-200K with simulated barge-in overlap and MUSAN speaker/noise interference | BLEU, sBERT (response quality); response latency (RL), response success rate (RSR); barge-in stop latency (SL) and stop success rate (SSR), latencies >1.5s counted as failures | IRAF improves multi-turn InstructS2S-200K under MUSAN speech interference by +4.12 BLEU (+42.7% relative) and +0.11 sBERT (+23.4% relative) over the noisy-augmentation baseline while preserving full-duplex interaction metrics. | Interference simulated (MUSAN) rather than real acoustic scenes; fixed 0.64s inter-turn pause and 1.5s latency cap; text-token LLM output (speech decoder separate); single interfering-speaker assumption. | high |
| jalturn2026 | 2026 | turn-taking detection model (joint acoustic-linguistic) | public multilingual STurn-v3 benchmark + in-house Japanese customer-service corpus; frozen SenseVoice + CPC encoders with cross-attention | hold/shift accuracy, F1, end-to-end latency | JAL-Turn achieves 93.27%/0.934 on STurn-v3 (36ms) and 92.03%/0.925 on in-house Japanese (38ms), beating LLM-based detectors (GPT-5.1: 85.52%, 1205ms) with over 5x lower latency, but smaller gains on the public benchmark (vs STurn-v3: +0.2% acc). | gains over STurn-v3 on the public set are marginal; in-house Japanese corpus is undisclosed; attribution shows heavy reliance on ASR (SenseVoice) features | high |
| japdst2025 | 2025 | method study: DST incorporated into Moshi (Japanese full-duplex task-oriented spoken dialogue) | JMultiWOZ benchmark corpus for Japanese task-oriented dialogue | dialogue state tracking accuracy, response generation quality | Incorporating DST into Moshi yields a multi-channel full-duplex task-oriented spoken dialogue model, evaluated on JMultiWOZ; DST and response generation improve over the plain full-duplex model, though abstract reports no numeric scores. | abstract-only; Japanese-only; JMultiWOZ scale not stated in abstract | moderate |
| japfd2025japanese | 2025 | J-Moshi: Moshi adapted to Japanese — Japanese GPT-2 SentencePiece tokenizer (32k), reinitialized text-vocabulary weights, frozen Mimi; pretraining on J-CHAT (mono, diarized into stereo) then fine-tuning on real stereo corpora plus multi-stream-TTS synthetic augmentation (J-Moshi-ext) | J-CHAT 68,892 h pretraining (128 V100, 8,880 steps, 36 h); fine-tune 344 h real stereo (5 corpora: Callhome, CSJ, Travel Agency, Casual, Consultation) + 602 h synthetic from 43,739 LLM-rewritten dialogues (J-Moshi-ext: 946 h total, synth WER 24.6%); 7B temporal transformer | Prompted dialogue continuation (709 samples): PPL (llm-jp-3.7B), human naturalness/meaningfulness 5-point (125 crowdworkers), turn-taking statistics vs English Moshi | J-Moshi beats dGSLM on naturalness 2.67 vs 2.44 and meaningfulness 2.19 vs 1.76 (J-Moshi-ext 2.30, showing TTS augmentation helps language capability), and generates Japanese-typical overlap (5.0 s/min vs English Moshi's 1.2 s/min, GT 8.1). | Scores far below re-synthesis (3.90/3.92) and ground truth (4.46/4.45); 88% PAD-token ratio in J-CHAT; pretraining data is diarized mono without natural overlap; Mimi left unadapted | high |
| liberating2026 | 2026 | Listen-Write-Speak (LWS): text-first tri-channel paradigm (continuous listening, visible free-form writing, parallel realtime speech) implemented purely via a Token Schema on a single autoregressive LLM — no architectural changes; two-stage teacher-LLM data pipeline with per-second causal annotations and interruption augmentation | MiniCPM-V architecture with Qwen3-8B backbone; 500K Chinese+English 1-second-unit training examples; frozen streaming Whisper encoder + TTS | Full-Duplex-Bench interaction, VoiceBench AlpacaEval score, writing-speaking consistency, URO-Bench (U/R/O) | LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. | Visible-writing output is an unusual product format; speaking handled by external TTS conditioned on LLM states; 500K examples is a moderate scale. | high |
| liu2024fullduplex | 2024 | Full-duplex generative dialogue system: 8B LLM aligned to a perception module, motor-function module, and a two-state neural FSM; LLM emits control tokens (respond/wait/interrupt) via next-token prediction on a serialized real-time dialogue view | 8B-parameter LLM (training data and alignment details not stated) | Average conversation response latency, % responses under 500ms, interruption precision rate (automatic evaluation simulating real interaction) | The system reduces average response latency by more than 3x vs LLM-based half-duplex systems, responds within 500ms in over 50% of interactions, and achieves 8% higher interruption precision than the best available commercial voice LLM. | No training-data or alignment details in abstract; two-state FSM is a coarse turn-taking model; evaluation simulated rather than with real users. | moderate |
| lldm2025 | 2025 | Semantic VAD dialogue manager: 0.5B LLM fine-tuned on full-duplex conversations to emit 4 control tokens (<\|C-L\|>, <\|S-S\|>, <\|S-L\|>, <\|C-S\|>) managing turn-switching/keeping for a modular SDS (AEC + acoustic VAD + ASR + semantic VAD + CDE LLM + TTS); CDE activated only when needed | Synthetic full-duplex text conversations generated via LLM API (Yuanbao), 200 topics x 10 personas; 0.5B DM; ~4,000-decision test set (no training-set size reported) | Control-token classification accuracy/F1 (confusion matrix) vs DuplexConv and RTTL-DG on user-state detection and barge-in intention analysis | The 0.5B semantic VAD achieves 97.85% overall decision accuracy (per-token F1 0.956-1.000), reliably distinguishing intentional vs unintentional barge-ins and detecting query completion during pauses/hesitations. | DM trained/evaluated on synthetic text only (no acoustic input to DM); no real-dialogue or end-to-end quality evaluation; ASR error propagation unmodeled; training-data scale undisclosed | moderate |
| llmcodec2026 | 2026 | LLM-Codec: neural audio codec trained with language-model objectives - Future Token Prediction (FTP, Medusa-style multi-step heads initialized from LLM head, inverse-distance weighting) and Semantic Alignment (SA, aligning codec features with chosen LLM layers); three-phase training (D-only warmup 0-10k steps, +FTP 10-12k, +FTP+SA 12-25k) | training details: 25k steps in 3 phases with frozen LLM backbone and Gumbel bridge; scale not quantified in text read | speech LM evaluation (incl. speaker-related scores), reconstruction quality (spectral fidelity; music/environmental domains vs AUV baseline), token predictability | Training a codec with LM objectives (multi-step future-token prediction + semantic alignment) improves downstream speech LM performance - especially speaker-related scores - and spectral fidelity as a byproduct, with music and environmental reconstruction comparable to the AUV baseline. | Authors list speech-centric supervision, frozen-LLM backbone constraints, evaluation coverage, and training overhead as limitations; exact quantitative gains not captured in the sections read. | moderate |
| lrbrain2026 | 2026 | Modular human-machine full-duplex dialogue system with 'left-right brain' division of labor and collaboration among specialized components; voice-to-voice interaction covering interruption and refusal scenarios in Chinese and English | Not stated in abstract | HumDial Challenge Track 2 (full-duplex interaction: interruptions, speech competition, real-time feedback, conversational pacing) | The system achieved first place in Track 2 of the HumDial Challenge for full-duplex interaction. | Abstract only; no architecture, data, or latency details; competition result without error analysis. | low |
| lta2023listen | 2023 | LTU: LLaMA-7B (Vicuna) + frozen audio encoder (AST) with trainable Q-Former-style connector + LoRA adapters (rank 8); autoregressive next-token prediction; perception-to-understanding curriculum; OpenAQA-5M (audio, question, answer) tuples | OpenAQA-5M: 1.9M closed-ended + 3.7M open-ended audio QA tuples from 8 mainstream datasets (AudioSet-Strong, AudioSet, etc.); only 4.2M trainable parameters (LoRA) on LLaMA-7B | audio classification/captioning accuracy; open-ended audio QA; mAP-type benchmarks vs CLAP | LTU outperforms the conventional audio-text CLAP model on all 8 audio classification benchmarks with 23.6% average relative improvement, and exhibits emerging audio reasoning/comprehension abilities (e.g., unanswerable-question detection) absent in prior audio models. | Text output only (no speech generation); audio understanding limited to ~10s clips; reasoning quality depends on LLM backbone; LoRA keeps most knowledge frozen. | high |
| m3duplex2026 | 2026 | benchmark (multi-turn, multilingual, multidomain) | English+Japanese, chat (MagicData ~10h) and multi-turn QA (TopiOCQA-based synthetic); models Moshi, PersonaPlex, J-Moshi, LLM-jp-Moshi | TOR, smooth turn-taking latency, barge-in stop latency, response/context relevance, QA accuracy under None/User/Full context | Teacher-forced Full context cut Moshi's task-oriented turn-taking latency from 2.453s to 0.430s (J-Moshi 0.933 to 0.326s) and raised PersonaPlex QA accuracy 0.180 to 0.306, while Japanese models underperformed English ones mainly on content, not timing. | only Moshi-family architectures evaluated (Full setting requires parallel-stream conditioning); synthetic spoken data for QA domain | high |
| ma2025vita15 | 2025 | Three-stage training (vision-language; audio input; audio output) with NAR+AR speech decoders over single-codebook TiCodec; no separate ASR/TTS | ~8B-class LLM; 110,000 hours internal ASR data + 3,000 hours TTS data for the speech decoder | Image/video/speech benchmarks; end-to-end response capability (no quantified duplex latency in text) | Progressive three-stage modality integration relieves speech-vision training conflicts and yields near real-time speech-to-speech interaction with comparable vision-language performance, without external ASR/TTS modules. | No dedicated full-duplex/interruption evaluation or latency numbers reported; duplex behavior relies on VITA-style parallel monitoring rather than native dual-stream modeling. | high |
| minicpm2026 | 2026 | Omni-Flow time-aligned full-duplex streaming framework; end-to-end token-level connections across encoders/LLM/speech decoder; LLM emits text (3-4 tok/s) and a 0.3B decoder emits S3 speech tokens; GRPO RL post-training | ~9B total (Qwen3-8B backbone, Whisper-Medium encoder); millions of hours unlabeled speech + studio-recorded spoken dialogue; 500K+ curated samples | Vision-language benchmarks, omni-modal understanding, speech generation, RTF/memory at inference, Full-Duplex results | MiniCPM-o 4.5 (9B) runs real-time full-duplex omni-modal interaction on edge devices (<12 GB RAM, RTF 0.21 at INT4 via llama.cpp-omni) while surpassing Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation quality. | Streaming speech occasionally unstable (mispronunciation, EN/ZH mixing); proactive behavior still simple; long-horizon real-world robustness unvalidated. | high |
| miniomni2024 | 2024 | Qwen2-0.5B LLM + Whisper-small encoder + SNAC 7-layer codec; text-instructed delayed parallel decoding (text + 7 audio layers per step); batch parallel decoding; 'Any Model Can Talk' three-stage training (modality alignment, adaptation, full fine-tune) | ~8,000h ASR data (LibriTTS 586h, VCTK 44h, M- LibriSpeech 8000h), 2M Open-Orca text QA, 1.5M Moss-002-sft synthesized speech QA, VoiceAssistant-400K GPT-4o-generated; 0.5B model on 8 A100s | LibriSpeech ASR WER; textQA/speechQA; streaming audio output capability; ASR WER 4.5 test-clean / 9.7 test-other | First open-source fully end-to-end streaming speech-in/speech-out conversational model: a 0.5B Qwen2 model with Whisper encoder and SNAC codec achieves LibriSpeech test-clean WER 4.5 (vs Whisper-small 3.4) while generating audio and text in parallel streams with minimal first-token delay, using the VoiceAssistant-400K dataset and 'Any Model Can Talk' adapters. | Authors report speech-modality reasoning is weaker than text reasoning (motivating batch parallel decoding); evaluation in v1 limited ('more experiments in next version'); no explicit duplex/interruption handling in this version. | high |
| miniomni22024 | 2024 | Qwen2-0.5B + CLIP ViT-B/32 vision encoder + Whisper-small audio encoder; text-audio delayed parallel decoding (7 SNAC codebook heads, vocab 181,120); three-stage training; command-based semantic interruption with frame-level irq/n-irq state tokens ('stop omni') | Same ASR mix as Mini-Omni (LibriTTS/VCTK/MLS ~8.6k h), Open-Orca 1.5-2M, Moss-002-sft speech QA, ALLaVA-4V 400k image QA, VoiceAssistant-400K | image caption, ASR, response-speed cases; duplex interruption behavior (irq token accuracy) | Extends Mini-Omni to vision+speech+text with a three-stage training pipeline and a command-based interruption mechanism where the model emits irq/n-irq tokens to stop its own speech stream on hearing 'stop omni' amid noise, arguing full-duplex training alone is 'not yet sufficiently stable'. | Paper explicitly states comprehensive vision/text experiments 'will be updated shortly'; interruption limited to a single command phrase; small 0.5B backbone; no quantitative duplex metrics reported. | moderate |
| minmo2025 | 2025 | Aligned multimodal model (Qwen2.5-7B + SenseVoice encoder + CosyVoice2-based AR streaming voice decoder, 5:15 text:speech token mix) with a separate Full-Duplex Predictor module | ~8B params; 1.4M hours speech data; multi-stage S2T/T2S/S2S/duplex alignment | Turn-taking positive-F1 @offset K, backchannel accuracy, system latency (ms) | MinMo reports ~600 ms theoretical / ~800 ms practical full-duplex latency (user turn-taking avg 250 ms; assistant turn-taking avg ~660 ms), ~99% positive-F1 at K=10 on the simulation set, 70-80% backchannel accuracy, and 98.4% instruction-following voice generation accuracy. | Paper admits duplex module still needs separate AEC and VAD modules; text LLM only LoRA-updated; S2S QA lags S2T; long-tail pronunciation errors. | high |
| moshi2024speech | 2024 | dual-stream (multi-stream) speech-to-speech transformer: 7B text LLM backbone (Helium) + temporal/depth RQ-transformer hierarchy; Mimi neural codec with semantic distillation into first RVQ level; Inner Monologue (time-aligned text tokens predicted as prefix to audio) | Helium 7B pretrained on 2.1T text tokens; audio pretraining on 7M hours unsupervised audio (Whisper-transcribed); duplex fine-tuning on 2000h Fisher stereo phone calls + 170h supervised multi-stream data + >20k h synthetic instruct speech | theoretical latency 160ms, ~200ms practical; streaming ASR/TTS quality; spoken QA accuracy; speech intelligibility/audio quality vs non-streaming baselines; dialogue statistics vs human | First real-time full-duplex spoken LLM: models user and agent audio as two parallel autoregressive streams (no speaker-turn segmentation) with 160ms theoretical / ~200ms practical latency, trained in 4 phases (unsupervised pretrain, simulated multi-stream post-train, Fisher fine-tune, synthetic instruct tuning), reporting speech quality and spoken-QA results competitive with non-streaming systems. | Authors note unconditioned streaming audio LMs struggle with intelligibility (mitigated by semantic-token distillation and Inner Monologue); latency measured on specific hardware; safety analyses limited to toxicity/regurgitation/voice consistency. | high |
| mtruplex2026 | 2026 | benchmark (multi-round full-duplex) | 100 evaluation samples x 10 rounds (Llama-Question/AdvBench reorganized); baselines Moshi, Freeze-Omni, VocalNet (HD), cascaded ASR+GPT-4o+ChatTTS | success rate and latency for smooth turn-taking/interruption/pause/background speech; GPT-4o dialogue quality; instruction-following success; safety refusal | Across rounds 1 to 1-10 Moshi's smooth turn-taking success drops 73.00% to 57.40% and its instruction-following 68.0% to 41.9%, while dialogue quality scores rank VocalNet (3.96) > Freeze-Omni (3.48) > Moshi (3.13), showing e2e FD-SLMs degrade with round count. | only two open-source FD-SLMs available as baselines; GPT-4o scores needed rescaling due conservatism | high |
| multimc2026 | 2026 | benchmark (multi-turn spoken dialogue on natural human interaction) | not stated in available text (ACL 2026 long paper; only citation metadata locally) | not stated in available text | Audio MultiChallenge evaluates spoken dialogue systems on multi-turn natural human interaction; concrete scale and results not available in local text. | only citation metadata available locally | low |
| nguyen2023textually | 2023 | TWIST: training SpeechLMs with warm-start from pretrained textual LMs instead of cold-start audio-only pretraining; analyzes speech tokenizer choice, pretrained model choice, and dataset size | Largest SpeechLM at the time in parameters and training data (exact sizes not in abstract); introduces spoken StoryCloze benchmarks | Automatic and human evaluations vs cold-start SpeechLM; spoken StoryCloze accuracy (no numbers in abstract) | TWIST outperforms cold-start SpeechLM across the board, with both model and dataset scale playing important roles (no numeric results in abstract). | No metrics in abstract; generation-only, no duplex input stream; English-focused evaluations. | moderate |
| ntpp2025ntpp | 2025 | Two-stage decoder-only SLM trained with Next-Token-Pair Prediction (NTPP) on time-aligned dual-channel speech: predicts the joint next token pair of both speakers, no VAD, VQ/RVQ tokenizers (40 tok/s, 4096 codebook) | Stage-1 pretraining ~140,000 h single-channel speech; stage-2 dual-channel Fisher; backbones LLaMA3.1-8B / Mistral-7B / Gemma-2-9B; 25-annotator MOS study | Turn-taking statistics deviations (IPU/pause/gap/overlap), MOS naturalness & meaningfulness (Fisher, CANDOR), interruption response, inference latency, speaker-swap robustness | NTPP (LLaMA-8B) tops human ratings (Naturalness 4.05-4.42 vs Moshi 3.90-3.95, dGSLM 3.85, SyncLLM 4.10) and keeps latency below the ~220 ms perceptual threshold across turn rounds via a single shared KV cache, whereas Moshi's two transformers cause linear latency growth. | Paper acknowledges scarcity of large-scale dual-channel data; interruption tests use GPT-4+ChatTTS synthetic scenarios; no reasoning/instruction-following evaluation | high |
| omni2025omniflatten | 2025 | OmniFlatten: E2E GPT model (Qwen2-0.5B) + CosyVoice semantic speech tokenizer; all speech/text streams flattened into one token sequence; multi-stage post-training: modality alignment (ASR+TTS), half-duplex dialogue (4 streams), full-duplex dialogue on chunked 3-stream then 2-stream data; simulated full-duplex dialogue data with barge-in scenarios and MUSAN noise | Qwen2-0.5B base; 2,000 hours of simulated multi-channel spoken dialogue data (1% val / 1% test) synthesized from Alpaca/Moss/BelleCN/ultraChat via CosyVoice | ASR WER/CER (LibriSpeech/Wenetspeech), TTS WER/CER, LLM-as-judge chat quality, assistant/user turn-taking Acc@K (1/5/10/25 tokens) and response time (ms) | OmniFlatten reaches assistant turn-taking Acc@K of 20.6/53.6/66.3/71.7% at 193ms average response time vs Moshi's 2.9/18.8/38.5/55.1% at 553ms, and user turn-taking 10.9/30.9/41.8/51.8% at 287ms vs Moshi's 0.0/6.2/14.8/45.7% at 753ms, with progressive gains from the half-duplex and 3-stream/2-stream stages. | Authors acknowledge 0.5B scale and 2,000h data limit chat quality; user turn-taking success within 25 tokens remains low for both models; no backchannel handling; conflicts between early-stop objective and speech-text alignment. | high |
| openfd2025agent | 2025 | Data-efficient framework converting any standard text LLM into a full-duplex end-to-end speech-to-speech model without extensive speech-text pretraining; open-source inference optimization + serving | Not stated in abstract | Full-duplex behavior, low-latency voice-agent operation (system demo; no quantitative metrics in abstract) | Claims any text LLM can be converted to simultaneously listen and speak with low latency and no extensive speech-text pretraining, lowering the barrier to building full-duplex voice agents. | No numbers in abstract; system demo + code release; verification would require reading the ASRU paper. | low |
| personaplex2026 | 2026 | PersonaPlex: Moshi-initialized duplex conversational speech model with hybrid system prompts (text role conditioning + voice-cloning speech sample); fine-tuned on synthetic role-conditioned dialogues; loss down-weighting (non-semantic audio tokens x0.02, padded text x0.3) | 2,250 h synthetic: 1,840 h customer service (105,410 dialogs) + 410 h QA (39,322 dialogs), TTS via Dia/Chatterbox from 26,296 voice samples; Moshi init (7B+); 24,576 steps, 8xA100, 6 h | DMOS naturalness (Full-Duplex-Bench + new Service-Duplex-Bench extension), WavLM-TDNN speaker similarity, FDB TOR/latency/JSD/GPT-4o score, Service-Duplex-Bench GPT-4o role adherence | PersonaPlex leads DMOS 3.90 vs Gemini 3.72 and Moshi 3.11, speaker similarity 0.57 vs <=0.10 for others, user-interruption latency 0.40 s vs Qwen2.5-Omni 2.74 s, and 4.48 on Service-Duplex-Bench (second only to Gemini 4.73). | Entirely synthetic training data; Service-Duplex-Bench uses single-turn probes rather than multi-turn role tests; inherits Moshi architecture constraints; benchmark extension not yet released | high |
| prompttt2025 | 2025 | turn-taking model study (PTTM robustness to noise) | PTTMs tested in clean and noisy conditions; audio-only vs multimodal (audio+visual) variants; ASR-derived transcriptions | hold/shift accuracy; 10dB music noise condition | PTTM hold/shift accuracy drops from 84% in clean speech to 52% in 10 dB music noise, and training with noisy data lets a multimodal PTTM reach 72% accuracy in that condition, though gains do not always generalize to unseen noise types. | abstract-only; noise types limited; ASR transcriptions usable only in clean conditions | moderate |
| pseudostereo2024 | 2024 | dGSLM-style dual-tower dialogue generative spoken LM; pseudo-stereo data pipeline (speaker diarization + SepFormer source separation + speaker verification) converts single-channel dialogue to two-channel; unit encoders compared across SSL foundation models (WavLM, data2vec, etc., incl. ASR-fine-tuned variants) | ~20k h scraped podcasts -> ~15.6k h pseudo-stereo dialogue; total training data expanded from 2,000 to 17,600 hours; 100h used for k-means (500 clusters) | turn-taking event statistics (IPU, gap, overlap, pause durations/occurrences vs ground truth), M-MOS semantic coherence, N-MOS naturalness | Pseudo-stereo data expanded training from 2,000 to 17,600 hours and improved M-MOS semantic coherence, while an ASR-fine-tuned foundation model gave the best units (scaling base SSL models alone hurt vocoder resynthesis); turn-taking behaviors were already capturable with Fisher-scale data. | Authors note N-MOS differences vs ground truth <=0.4 are dominated by vocoder artifacts; overlap statistics show limited gains from pseudo-stereo data; pipeline errors from diarization/separation. | high |
| qwen2audio2024 | 2024 | Qwen-7B LLM + audio encoder initialized from Whisper-large-v3; three-stage training: pretraining with natural-language prompts instead of hierarchical tags, supervised fine-tuning with two jointly-trained modes (voice chat, audio analysis), then DPO preference optimization | Qwen-7B backbone; pretraining and SFT data hours not quantified in the text read; encoder from Whisper-large-v3 | ASR (WER/CER), speech-to-text translation (BLEU), SER, vocal sound classification, AIR-Bench audio chat accuracy | Qwen2-Audio outperforms previous audio-LLM SOTAs including Gemini-1.5-pro on AIR-Bench audio-centric instruction-following (top chat accuracy 0.9392 reported), with no system prompts needed to switch between voice-chat and audio-analysis modes, and DPO improving factuality. | Text-only output (external TTS needed for speech); voice chat is turn-based, no duplex/streaming generation; evaluation focuses on understanding. | moderate |
| qwen3omni2025 | 2025 | Single unified Thinker-Talker MoE model; multi-codebook (MTP) Talker + causal ConvNet Code2Wav; AuT audio encoder, TM-RoPE, chunked prefill | 30B-A3B (Thinker) + 0.65B AuT encoder trained on 20M hours supervised audio; audio code rate 12.5 Hz | Benchmark SOTA counts; end-to-end first-packet latency (ms) | Qwen3-Omni attains open-source SOTA on 32 of 36 audio/audio-visual benchmarks (overall SOTA on 22, beating Gemini-2.5-Pro/Seed-ASR/GPT-4o-Transcribe) with a theoretical 234 ms end-to-end first-packet latency from single-frame multi-codebook streaming synthesis. | Not truly full-duplex: turn-taking still relies on external VAD; report optimizes first-packet latency/streaming, not barge-in or overlap handling; Talker now text-decoupled from Thinker. | high |
| qwenomni2025 | 2025 | Thinker-Talker architecture (LLM text generation + dual-track AR Talker audio tokens from Thinker hidden states); block-wise audio/video encoders, TMRoPE position embedding, sliding-window DiT streaming vocoder | Comparable to similarly-sized Qwen2.5-VL (size class not quantified in abstract) | Omni-Bench, MMLU/GSM8K speech instruction following, speech robustness/naturalness | Qwen2.5-Omni matches similarly-sized Qwen2.5-VL, outperforms Qwen2-Audio, achieves SOTA on Omni-Bench, and performs end-to-end speech instruction following comparable to its text-input performance on MMLU/GSM8K. | Abstract only; describes streaming half-duplex generation, not native full-duplex overlap/interruption handling. | moderate |
| realtalk2024 | 2024 | Face-to-Face spoken dialogue model: audio-visual input and output without intermediate text, built by adapting a textually pretrained LLM via speech-text joint pretraining; introduces MultiDialog corpus | MultiDialog: 340 hours, ~9,000 audio-visual dialogues based on TopicalChat with emotion annotations | Automatic and human evaluation of face-to-face conversation effectiveness (no numeric results in abstract) | The text-LLM + joint speech-text pretraining approach yields a valid face-to-face dialogue model on the new 340h MultiDialog corpus, but no quantitative results are reported in the abstract. | Abstract contains no metrics; scripted turn-based recordings rather than overlapping full-duplex interaction; audio-visual scope beyond audio-only duplex. | moderate |
| realtalkcn2026 | 2026 | benchmark + dataset (Chinese multi-turn multi-domain speech-text TOD, cross-modal chat) | 5.4k dialogues, 60K utterances, 150h, 58 domains, 55 intents, 115 slots; disfluency-annotated; baselines Qwen2-Audio-7B, Baichuan-Audio, GLM-4-Voice-9B, MiniCPM-o, Baichuan-Omni-1.5, Qwen2.5-Omni-7B, Pipeline (Whisper-large-v3+GPT-4o) | intent accuracy (PANDA), slot F1/JGA, GPT-4o-based chat scores, cross-modal chat and disfluency robustness | On the chat task the text Pipeline (Whisper+GPT-4o) scores 9.00 avg vs best end-to-end speech model GLM-4-Voice at 8.37, showing speech-native LLMs still trail pipelined text backbones on Chinese TOD; intent/slot numbers reported per subset. | task-oriented and turn-based (no continuous barge-in/overlap evaluation); scores are GPT-4-based automated judgments; Chinese-only | high |
| routing2026 | 2026 | Controlled comparison of two user-stream routing strategies for extending a text-only LLM into full-duplex spoken dialogue under a shared pipeline: channel fusion (CF-Duplex, user stream injected into LLM input) vs cross-attention routing (XA-Duplex, user stream as external KV memory) | Matched CF/XA model variants with a staged training curriculum and tailored interruption data | Spoken QA accuracy, Full-Duplex-Bench interruption/backchannel metrics, coherence of generation under missed interruptions | Channel fusion gives stronger semantic grounding and better spoken QA, but under user interruptions it is more vulnerable to context corruption (incoherent continuation when the model fails to stop), while cross-attention routing preserves the generation context and is more robust to this failure mode. | I read introduction, related work, and method overview but not the result tables in full, so quantitative scores are not reported here; conclusions are the paper's stated tradeoff. | moderate |
| salm2025duplex | 2025 | SALM-Duplex: duplex S2S LM with continuous user input stream (100M streaming CTC encoder, 80ms frames) + codec agent output stream (4 codebooks at 12.5Hz) fused by channel addition into a text LLM (TinyLlama-1.1B-chat); predicts text and speech channels simultaneously with turn-level alignment; partially causal codec allows 0.6kbps and codec personalization; no speech pretraining required | ~26.7k h synthetic duplex data: ASR-QA 20k h (8k public + 12k in-house ASR labels -> Mixtral-8x22B QA -> TTS), MS MARCO 0.2k, Alpaca 0.2k, Internal SFT 3k (multi-turn), UltraChat 3k, Topic 0.3k; barge-in created by cutting agent speech with 0.64s residual | barge-in success rate and false-alarm rate, 1st response latency (s), UTMOS, GPT-score reasoning (0-10) on UltraChat/Roleplay/Topic/ASR-QA/Alpaca | SALM-Duplex (TinyLlama-1.1B) achieves barge-in success 83.0% (UltraChat) and 94.5% (Impatient) vs Moshi's 56.0%/55.1% with 0% false alarms, 1st-response latency 0.52s/0.69s vs 0.63s/0.81s, UTMOS 4.3/4.0 vs 3.9/3.8, and higher GPT reasoning scores (e.g., 7.8 vs Moshi on ASR-QA), without any speech pretraining. | Small 1.1B backbone limits reasoning ceiling; data almost entirely synthetic (TTS); barge-in simulated by simple cutoff; codec personalization evaluated only for single agent voice. | high |
| salmonn2023 | 2023 | dual encoders (Whisper-Large-v2 speech + BEATs audio) concatenated at 50Hz, window-level Q-Former connection module, Vicuna-13B LLM with LoRA adapters; three-stage training (pretrain on ASR+captioning, instruction tuning on 15 tasks, few-shot activation tuning) | Vicuna-13B; only ~33M trainable parameters (Q-Former + LoRA, ~0.24% of model); pretraining data: LibriSpeech 960h + GigaSpeech-M 1000h + WavCaps 2800h + music corpora (~400h+), plus instruction-tuning task data | accuracy/following-rate across 15 speech/audio/music tasks at 3 levels; emergent abilities (speech translation to untrained languages, spoken QA, storytelling, speech-audio co-reasoning) | SALMONN achieves competitive performance on 15 speech/audio/music understanding tasks with only ~33M trainable parameters on a frozen Vicuna-13B, and gains emergent cross-modal abilities (e.g., audio-based storytelling, speech-audio co-reasoning) via a cheap few-shot activation tuning stage that also alleviates task over-fitting. | Text-only output (no speech generation); no streaming or duplex interaction; activation tuning requires curated few-shot data. | high |
| scaling2024speech | 2024 | Scaling-law empirical study of Speech Language Models (audio-only, no textual resources); correlates pretraining loss with downstream syntactic/semantic performance to predict compute needed for text-LLM-level English proficiency | Multiple SLM sizes trained from raw audio; scale estimates projected for matching text-LLM proficiency (exact sizes not in abstract) | Pretraining loss and downstream syntactic/semantic benchmark performance (scaling exponents) | Linguistic performance of SLMs scales up to three orders of magnitude more slowly than text-based LLMs, though a strong loss-to-performance correlation makes the gap predictable. | No quantitative model sizes or dataset hours in abstract; English-only; weak syntax/semantics of current SLMs; not full-duplex specific. | moderate |
| sdfactory2025 | 2025 | data generation toolkit (synthetic speech dialogue) | released datasets SDF_en 3,168 dialogues/437,351s and SDF_zh 1,005/86,415s; evaluated against DailyTalk (2,541 dialogues); CosyVoice2 and Fish-TTS; Llama-3.2-1B/3.1-8B/3.3-70B | content consistency/coherence/naturalness (GPT-4o), UTMOS speech quality, WER intelligibility, x-vector speaker consistency | Generated dialogues match or exceed human recordings (CosyVoice2 speech quality 3.38 vs human 3.28; 8B-model naturalness 92.3 vs DailyTalk 90.3) with >30 speakers and lower production cost, though the authors note large-model compute needs and LLM cultural bias as limitations. | scripted turn-taking only — does not generate genuine overlap/barge-in dynamics; stated compute demands and cultural representation bias | high |
| seminterrupt2026 | 2026 | benchmark + metric + model (semantic interruption detection) | SID-Bench: 3,200 conversational instances (bilingual CN/EN, 500 uninterrupted, 1,100 interruptions, 500 noise/silence), ~10h curated audio; baselines FSMN-VAD, Freeze-Omni(Silero), FireRedChat(pVAD), Moshi | False Interruption Rate (FIR), Interruption Response Latency (IRL), composite Average Penalty Time (APT) | The proposed LLM-based SID model achieves 0.711s APT — a nearly threefold reduction over the best baseline Freeze-Omni (2.129s), while Moshi is robust to backchannels (FIR 0.118) but catastrophically slow (IRL 2.517s, APT 3.192s). | labels derive from LLM-annotated semantic breakpoints + forced alignment; proposed SID model itself is not fully specified open-source; synthetic-free but curated vendor data | high |
| sft2024hours | 2024 | KE-Omni seamless speech LLM (GPT-4o-style real-time interaction) trained on Ke-SpeechChat synthetic speech dialogue dataset; abstract-level details only | Ke-SpeechChat: 7 million Chinese+English conversations, 42,002 speakers, over 60,000 hours of synthetic speech interaction data | not reported in abstract (real-time interaction capability claimed) | KE-Omni scales supervised fine-tuning with over 60,000 hours of synthetic bilingual speech dialogue data (7M conversations, 42,002 speakers) to build a seamless real-time speech interaction LLM, the largest such SFT data recipe reported in this batch. | Abstract reports no quantitative evaluation (no latency/WER/MOS numbers); synthetic-data-only training risks distribution gaps; Chinese-focused. | moderate |
| si2023spokenwoz | 2023 | large-scale speech-text task-oriented dialogue benchmark: 8 domains, human-to-human spoken dialogues with word-by-word processing and reasoning-in-spoken-language characteristics; tasks include cross-turn slot and reasoning slot detection, DST, response generation | 203k turns, 5.7k dialogues, 249 hours of audio (paid $30k for collection); 4,200 dialogues in training set | joint goal accuracy (JGA) for DST; user-request completion rate for end-to-end response generation | SpokenWOZ shows current models lag far behind text TOD: the best dual-modal dialogue state tracker (SPACE+WavLM aligned) reaches only 25.65% joint goal accuracy and the SOTA end-to-end model completes the user request in only 52.1% of dialogues (LLM baselines like ChatGPT also far below text-level performance). | Authors note dataset focus on English task-oriented calls; ASR noise and spoken disfluencies make it harder than written WOZ; LLM analysis shows substantial room for improvement. | high |
| sidon2026 | 2026 | data-restoration model (joint restoration + separation of monaural dialogue audio) | English, multilingual, and in-the-wild dialogue datasets; SSL-VAE latent compression + diffusion-based latent predictor | speech intelligibility, separation quality, inference speed vs baseline | DialogueSidon substantially improves intelligibility and separation quality of degraded monaural two-speaker dialogue over the baseline while achieving much faster inference; no numeric scores in the local text. | quantitative gains not captured in available text; relies on SSL feature space which may limit generalization to extreme degradation | moderate |
| silent2026 | 2026 | FLAIR: latent (embedding-level) reasoning while listening — recursive latent-state feeding with an ELBO objective and a non-causal Global-aware Expert whose posterior is distilled into the causal model via KL; no explicit reasoning tokens, no extra inference latency | Qwen2.5-7B-Instruct backbone + Parakeet 600M streaming speech encoder; trained on 64 A800 GPUs with interleaved pretraining + latent-reasoning SFT | QA accuracy (LlamaQ, WebQ, TriviaQA, SDQA, MMSU, OpenbookQA), VoiceBench AlpacaEval, turn-taking/barge-in latency and success rate, MOS | Think-while-listening via latent reasoning improves response quality across nearly all QA benchmarks (largest gains on reasoning-heavy MMSU/OpenbookQA) while keeping turn-taking and barge-in performance comparable to non-thinking full-duplex baselines with zero added inference latency. | Latent thoughts are uninterpretable; gains smaller on knowledge-retrieval tasks; requires the two-model ELBO training setup (expert + causal student). | high |
| slamming2025 | 2025 | Slam: training recipe for SLMs on a single academic GPU in 24 hours - empirical analysis of model initialization, architecture, synthetic training data, and preference optimization with synthetic data | Single-GPU, 24-hour training budget (model sizes and dataset hours not in abstract) | SLM benchmark performance vs leading SLMs; comparison against SLM scaling-law compute-optimal predictions (no numbers in abstract) | The recipe reaches results on par with leading SLMs at a fraction of compute cost, far outperforming predicted compute-optimal performance under SLM scaling laws. | No concrete metrics in abstract; single-GPU budget caps scale; heavy reliance on synthetic data and preference optimization details not in abstract. | moderate |
| slamomni2025 | 2025 | SLAM-Omni: single-stage end-to-end voice interaction system; grouped speech semantic tokens shorten token sequences (faster train/inference), speaker info decoupled to vocoder for zero-shot timbre control, historical text prompting compresses dialogue history | 15 hours of training on 4 GPUs with limited data (model size not stated); larger multilingual/multi-turn datasets for further validation | Voice interaction quality and latency vs prior same-scale models (no numbers in abstract) | SLAM-Omni outperforms prior models of similar scale using only 15 GPU-hours-equivalent training (15h on 4 GPUs), claimed first spoken dialogue system competitive with single-stage training and no TTS/ASR pretraining. | No metrics in abstract; very small training budget may limit generalization; grouped semantic tokens may sacrifice prosodic detail. | moderate |
| smarc2026 | 2026 | S-MARC: streaming causal hierarchical conversational behavior modeling — behavior perceiver (high-level communicative functions + low-level interaction behaviors) with continuously updated graph-of-thought reasoning and justifications | ConversationGoT-120h: 120-hour benchmark corpus of real and synthetic duplex dialogues with causal 1-second behavior annotations and rationale labels | Behavior detection (F1/accuracy vs baselines), human-model agreement, rationale quality rubrics | S-MARC achieves robust behavior detection on synthetic and real duplex dialogues with interpretable, causally-ordered reasoning chains, establishing a benchmark foundation for conversational reasoning in full-duplex systems. | Analysis/perception framework rather than an end-to-end spoken model; 120h corpus is modest scale; annotation quality depends on LLM-based rationales. | moderate |
| sommelier2026 | 2026 | data processing pipeline (scalable open multi-turn audio pre-processing for full-duplex SLMs) | pipeline for multi-speaker conversational data; addresses overlap and back-channeling, diarization errors, ASR hallucinations | pipeline robustness/scalability (qualitative in abstract) | Proposes an open-source scalable data processing pipeline for full-duplex SLM training data, targeting overlapping speech and back-channels while mitigating diarization errors and ASR hallucinations in existing single-speaker-limited corpora. | abstract-only; no quantitative evaluation of pipeline output in abstract | moderate |
| soulx2026 | 2026 | Plug-and-play streaming state prediction module unifying VAD+ASR+turn detection with text-guided (ASR-assisted) streaming prediction; frozen GLM-4-Voice tokenizer + Qwen3-0.6B LLM; 3-stage training (non-streaming ASR, streaming ASR, state-prediction SFT) | ~47K h Mandarin + ~31K h English ASR corpora; Fisher (thousand-hour) + ~10K-h in-house Mandarin state-annotated data; LoRA r=32 | State-prediction ACC, theoretical/inference latency (ms), Full-Duplex-Bench metrics (TOR, RL, SL, RsR, RpR) | SoulX-Duplug achieves a theoretical average streaming state-prediction latency of 240 ms (80 ms chunk-internal + 160 ms chunk), and a modular FD-SDS built on it outperforms existing full-duplex models on overall turn management and latency. | Modular design decouples control from generation (backbone Qwen2.5-7B + IndexTTS external); inference uses teacher-forced external ASR; benchmark subsets are TTS-synthesized. | high |
| sovabench2025 | 2025 | benchmark (speech LLM voice-assistant generation quality) | 8 tasks: 6,840 knowledge QA, 2,619 spoken QA, plus recognition/generation subsets; models Mini-Omni, LLaMA-Omni, Moshi, Freeze-Omni, GLM-4-Voice, Mini-omni2, cascades | GPTEval accuracy, WER, semantic/acoustic generation quality, text-speech consistency | Cascade systems retain 58.98% knowledge-QA accuracy vs best speech LLM Freeze-Omni at 27.27%, and larger LLM backbones (7B/8B) beat 0.5B models on knowledge, understanding, and semantic generation quality. | turn-based QA/instruction evaluation — does not evaluate simultaneous listening-speaking or overlap handling | high |
| speechgpt2023 | 2023 | SpeechGPT: LLM with intrinsic cross-modal conversational abilities using discrete speech representations; three-stage training - modality-adaptation pretraining, cross-modal instruction fine-tuning, chain-of-modality instruction fine-tuning; built on SpeechInstruct dataset | SpeechInstruct: large-scale cross-modal speech instruction dataset (scale not in abstract); model size not stated | Multimodal instruction-following capability (qualitative demos; no metrics in abstract) | SpeechGPT shows impressive capacity to follow multimodal human instructions, enabling cross-modal knowledge transfer within a single model (no quantitative results in abstract). | No numbers in abstract; turn-based conversation rather than full-duplex streaming; discrete-token speech generation; no interruption/overlap handling. | moderate |
| styles2024speaking | 2024 | StyleTalk dataset (same sentence in different speaking styles with different responses) + Spoken-LLM: frozen Llama 2-Chat 7B with LoRA + emotion2vec speech style encoder + lightweight connector; two-stage training; response conditioned on speaking style | StyleTalk: 1,878 dialogue sets / 1,986 samples (human-filtered train), 486 sets / 981 samples eval, plus 5,777 sets / 16,472 samples unfiltered LLM-generated augmentation; Llama 2-Chat 7B | BLEU, ROUGE-L, METEOR, BERTScore, self-BLEU (style-driven response diversity); human A/B evaluation | Spoken-LLM trained on StyleTalk learns to produce different responses for identical sentences spoken in different styles (text-only baselines cannot), with higher response diversity (self-BLEU) and better lexical/semantic scores than text-only and prior speech LLM methods. | Responses generated via style attributes + expressive TTS rather than direct speech-to-speech; dataset limited to ~2k curated samples; English-centric; turn-based. | high |
| survey2025speechlm | 2025 | survey (speech language models broadly) | systematizes SpeechLM components (tokenizer/LM/vocoder), training recipes (pre-train, instruction-tune, post-alignment), and interaction paradigms incl. multi-sequence (duplex) alignment examples (Llama-Omni, Mini-Omni, Moshi) | categorizes objective metrics (ASR-based, quality) and human subjective evaluation | Synthesizes SpeechLM training recipes and notes training a SpeechLM is significantly harder than a TextLM since speech requires learning spoken-language rules from scratch, and alignment of text and speech representations is effective but multi-sequence (duplex) alignment adds modeling complexity. | broad survey with limited depth on full-duplex specifics; no new experiments | high |
| sync2024llms | 2024 | Llama3-8B LLM expanded with 501 HuBERT speech tokens (25Hz) + [S0]/[S1] speaker tags; interleaved chunked dual-speaker token sequences with periodic synchronization tokens; deduplication + interpolation of HuBERT tokens; three-stage training (turn-based SFT on synthetic speech, no-overlap full-duplex, real spoken dialogue) | 212k h synthetic spoken dialogue (193k h SFT + 20k h dialogue) generated via Bark TTS from text data + 1,927h real Fisher spoken dialogue; Llama3-8B | MOS meaningfulness (M-MOS) and naturalness (N-MOS) vs dGSLM; turn-taking event correlation (IPU/pause/fto, Pearson r); two-agent full-duplex simulation under up to 240ms Internet latency | Synchronous LLM (SyncLLM) with wall-clock synchronization tokens and 160-240ms chunking achieves +2.2 MOS meaningfulness over dGSLM on Fisher while keeping naturalness, and sustains coherent two-agent full-duplex dialogue under simulated Internet latencies up to 240ms using only ~2k h of real spoken dialogue data. | Authors note audio reconstruction errors from token deduplication/interpolation heuristics; chunk sizes constrain timing granularity; evaluation via simulated agent-agent interaction rather than human users. | high |
| syncfd2026 | 2026 | Analysis study: simulated full-duplex dialogues between two Moshi instances connected over a token-level audio routing channel (noise and PAD-bias manipulated); synchronization via linear CKA across temporal lags; anticipatory turn-taking probed with causal LSTM classifiers | 2,880 simulated conversations (~80 h audio): 4 noise levels x 2 checkpoints x 2 pAD-bias levels; official Moshi (agent + client) checkpoints | CKA similarity vs temporal lag; end-of-IPU and Hold-vs-Non-Hold probe accuracy vs chance; IPU boundary timing | Moshi-Moshi dialogues show strong representational synchronization peaking near zero lag that degrades as channel noise rises, and delayed internal activations encode anticipatory turn-taking information decodable above chance for both speaker- and listener-side probes. | Two-instance simulated dialogues are an artificial proxy for human conversation; probe accuracy numbers fall in result tables not captured in the read portion. | moderate |
| synchronsurvey2025 | 2025 | survey (full-duplex spoken language models) | comparative analysis across ~7 mainstream FD-SLMs (dGSLM, Moshi, SyncLLM, OmniFlatten, Freeze-Omni, MinMo, Mini-Omni2, VITA-1.5, FlexDuo) | formal duplex definitions; four-pillar benchmark taxonomy (Temporal Dynamics, Behavioral Arbitration, Semantic Coherence, Acoustic Performance) | Surveys FD-SLMs under an Engineered-vs-Learned Synchronization taxonomy and identifies synchronous data scarcity, architectural divergence, and evaluation gaps, with a real-time constraint of <200ms compute per output token. | survey-level analysis without new benchmark results; taxonomy choices are author-defined | high |
| triadic2025vap | 2025 | turn-taking model (VAP for triadic multi-party) | Japanese triadic conversation dataset, multiple models trained; first VAP extension beyond dyads | future voice-activity prediction accuracy per speaker | VAP models trained on triadic conversation outperform the dyadic baseline for all tested configurations, establishing that VAP can predict turn-taking in triadic multi-party dialogue. | abstract-only; no numeric accuracies in available text; Japanese-only dataset | moderate |
| ttreview2025 | 2025 | review of turn-taking modelling in conversational systems: synthesizes voice activity (VA), turn floor offsets (TFO), VAD and inter-pausal units (IPUs), hold/shift concepts, predictive turn-taking, and computational heuristics/models | n/a (review) | n/a (discusses TFO timing vs minimal human reaction times, cross-linguistic variability) | Review concludes that turn transitions align closely with minimal human reaction times (TFOs), that simplistic silence-based heuristics reduce interruptions but risk unnatural delays, and that human-like responsiveness requires integrating prosodic, verbal, visual, and predictive (anticipatory) signals. | Narrative review; no standardized quantitative benchmark across systems. | high |
| tvoice2026 | 2026 | benchmark (grounded full-duplex voice agents, task completion) | 278 grounded tasks across real-world domains; voice user simulator with accents/noise; agent family + GPT-5 baseline | task completion pass@1, voice interaction quality, text-to-voice capability retention | On 278 tasks GPT-5 (reasoning) reaches 85% while voice agents only achieve 31-51% clean and 26-38% under realistic noise/accent conditions (retaining 30-45% of text capability), with 79-90% of failures attributed to agent behavior. | simulated user, not real humans; simulation decoupled from wall-clock time; accent effect highly provider-specific (xAI loses 38%, Google nearly none) | high |
| uaf2026 | 2026 | Unified audio front-end LLM: VAD, speaker recognition, ASR, turn-taking detection, and QA reformulated as one streaming sequence-prediction task with control state tokens (<SIL>/<TALK>/<Complete>/<InComplete>/<Interrupt>/<Backchannel>); 600 ms chunks + reference-audio speaker anchoring; LoRA on Qwen3-Omni thinker | 6,000 h (Stage I VAD/SR/ASR) + 1,000 h (Stage II TD/QA), trained on a synthetic multi-talker pipeline with echo, noise (0-20 dB SNR), and barge-in simulation | WER, VAD accuracy/precision/recall/F1, turn-taking detection accuracy, latency | UAF achieves 2.43 WER on AISHELL-2 and 13.75 on a challenging real-recorded Online-test (beating Qwen3-Omni-30B-A3B), with LoRA fine-tuning matching full fine-tuning within 0.1 WER on standard ASR sets. | Front-end only — speech generation/back-end still external; evaluations are Mandarin-centric; state-token taxonomy fixed at 6 states. | high |
| unitagent2026 | 2026 | Train-free semi-cascaded full-duplex system: decomposes dialogue into minimal conversational units; Qwen3-Omni MLLM makes unit-level turn-transition decisions directly from audio (replacing ASR-to-LLM cascade), with Silero VAD, CAM++ speaker verification, Paraformer ASR context, IndexTTS 1.5 streaming TTS | HumDial challenge (ICASSP 2026) dataset; 2x A100; plug-and-play (no duplex-specific training) | Full-Duplex-Bench v1.5-style: first-response delay, interruption rejection score, total delay, total score (dev + test) | Ranks 2nd in the HumDial Challenge Track 2 (Full-Duplex Interaction) with test total delay 1.632 s, score 89.7, and interruption rejection 57.8 vs baseline 2.436 s / 45.6. | Challenge report with limited analysis; performance bounded by frozen components; still needs auxiliary VAD/ASR; no comparison on semantic quality or reasoning | moderate |
| urob2025bench | 2025 | benchmark (comprehensive end-to-end spoken dialogue models) | not stated in available text (Findings of EMNLP 2025; only citation metadata locally) | not stated in available text | URO-Bench is proposed as a comprehensive evaluation for end-to-end spoken dialogue models (understanding, reasoning, and generation), per the paper's framing; local text has no numbers. | only citation metadata available locally; cannot verify scale or metrics | low |
| vita2024 | 2024 | Mixtral 8x7B SMoE LLM (vocab expanded 32,000->51,747 for Chinese, 5M synthetic bilingual corpus) + audio/vision encoders; three-stage training (LLM instruction tuning, multimodal alignment, multimodal instruction tuning); state tokens for query-type detection; duplex deployment with two simultaneously-running models (generation + monitoring) for audio interrupt and non-awakening interaction | Mixtral 8x7B (47B total / ~13B active per token); 5M synthetic bilingual text corpus; multimodal alignment and instruction data not quantified in text read | unimodal/multimodal benchmarks (e.g., ASR WER 8.14 test-clean / 18.41 test-other reported by OmniFlatten for VITA); interaction capability (interrupt handling) via case studies | VITA is the first open-source MLLM processing video/image/text/audio with an interactive duplex scheme - two model instances swap generation/monitoring roles so the system pauses generation and answers the latest query on user interruption, with state tokens enabling non-awakening interaction. | Speech output requires external TTS (not end-to-end); duplex achieved by system-level duplication rather than in-model dual-stream modeling; interruption is utterance-level (no backchanneling/overlap modeling). | high |
| vocalnet2025 | 2025 | VocalNet: first application of multi-token prediction (MTP) to speech LLMs - scalable, model-agnostic framework for real-time voice interaction replacing standard next-token prediction | VocalNet-1B and VocalNet-8B; trained with limited training data (hours not specified) | Generation speed and quality vs NTP baselines and open-source speech LLMs (no numbers in abstract) | MTP delivers simultaneous gains in generation speed and quality: VocalNet performs on par with mainstream Omni LLMs despite limited training data and significantly surpasses existing open-source speech LLMs. | No metrics in abstract; speed/quality tradeoffs and benchmark scope not detailed; full-duplex interaction behavior not evidenced from abstract. | moderate |
| voicebench2026 | 2026 | benchmark (LLM-based voice assistants, turn-based) | 6,783 synthetic and real spoken instructions, 8 tasks, diverse speakers/accents/reverberation/mispronunciations | task accuracy across general knowledge, instruction following, safety compliance | Introduces the first benchmark purpose-built for LLM-based voice assistants and its experiments expose limitations of current voice-assistant models under speaker, environmental, and content variations. | abstract-only; instruction-style single-turn evaluation, no overlap/barge-in assessment | moderate |
| voiceblender2025 | 2025 | Single-stage joint speech-text SFT on LoRA of an LLM backbone, mixing text-only SFT with speech recognition/translation, speech-based QA, and mixed-modal SFT data (vs multi-stage pipelines of prior SpeechLMs) | 3B-parameter model (vs prior 7B/13B SpeechLMs); data scale not stated in abstract | Speech benchmarks (ASR/translation/QA) plus preservation of text-only task performance (no numbers in abstract) | The 3B model outperforms prior 7B/13B SpeechLMs on speech benchmarks while preserving text-only capabilities, with emergent multi-turn and mixed-modal handling of unseen prompts. | No metrics in abstract; LoRA on 3B backbone may limit capacity for real-time duplex generation; multi-turn but not demonstrated true overlapping-speech interaction. | moderate |
| wavchat2024 | 2024 | survey of spoken dialogue models: chronological taxonomy into cascaded vs end-to-end paradigms; reviews speech representation (semantic vs acoustic, continuous vs discrete, single vs multi-layer quantizer), training paradigms (multi-stage alignment, SFT, preference optimization/RL), streaming, duplex and interaction technology, plus datasets/evaluation/benchmarks | n/a (survey) | n/a (survey; catalogs metrics: latency, WER, MOS/UTMOS, benchmarks) | Survey organizes spoken dialogue systems into cascaded vs end-to-end paradigms and identifies streaming, duplex (simultaneous listen+speak), and interaction as core capabilities, documenting that E2E duplex models trade conversational quality for latency and that turn-taking/duplex evaluation is still immature. | Survey snapshot (Nov 2024); qualitative synthesis rather than controlled experiments; coverage of 2025-2026 works limited. | moderate |
| wu2025wavtokenizer | 2025 | WavTokenizer: VQ-GAN acoustic codec with broader VQ space (16,384-codebook single quantizer, k-means init), extended context windows, improved attention, multi-scale discriminator, inverse-Fourier-transform decoder; single quantizer with temporal downsampling | trained on ~8K hours (LibriTTS, VCTK, Common Voice subset, music/audio); 24kHz output at 40 or 75 tokens/s | UTMOS (automatic MOS), objective reconstruction metrics on LibriTTS test-clean/test-other and LJSpeech; semantic-content tests, VQ utilization, generative-model adaptability | WavTokenizer reconstructs 24kHz speech/music/audio with SOTA UTMOS using only 40 or 75 tokens per second from a single quantizer (extreme compression), while containing richer semantic information than prior acoustic codecs and improving downstream generative-model performance. | Authors note codebook utilization limits (increasing training data does not raise utilization); single-quantizer design trades fine acoustic detail at very low rates; reconstruction speed/quality tradeoffs at 16/48kHz variants. | high |
| xtalk2025 | 2025 | X-Talk: open-source event-driven modular cascaded S2S framework (streaming frontend with enhancement/VAD, speech understanding incl. ASR/emotion/speaker, LLM agent with RAG+tool use, TTS) with buffered statement mechanism and async parallel modules for full-duplex interaction | No training: integrates off-the-shelf models (SenseVoice/Paraformer ASR, Qwen3-8B/30B/80B LLMs, IndexTTS 1.5/2, CosyVoice TTS); latency measured on 4xRTX 4090 for 5-60 s inputs | End-to-end latency (ms) across ASR/LLM/TTS combinations; sub-second streaming criterion; perceived responsiveness; false-interruption tolerance rules | Default config (SenseVoice streaming + Qwen3-30B + IndexTTS 1.5) yields 284-610 ms end-to-end latency for 5-60 s inputs, sub-second for all streaming-ASR configs, vs 3.7 s+ when ASR is offline. | No learned turn-taking — dynamics governed by VAD (500 ms endpoint) and rules; no standard full-duplex benchmark scores (FDB/latency-on-interruption); paralinguistic modules are non-blocking side channels rather than integrated decisions | moderate |
| xu2025llamaomni | 2025 | Whisper-large-v3 encoder (frozen) + Llama-3.1-8B-Instruct LLM + 2-layer speech adaptor + NAR streaming CTC speech decoder (425M params) + vocoder; simultaneous text and speech generation from speech input; two-stage training (S2T alignment then S2S) | InstructS2S-200K dataset (200K speech instructions + responses synthesized from text instruction data); speech decoder trained on ~1K h; total training <3 days on 4 GPUs | ChatGPT Score (content), ASR-WER (text-speech alignment), UTMOS, latency (ms), WPS; human evaluation (helpfulness/naturalness) | Latency as low as 236ms (chunk size 10; abstract claims 226ms) vs GPT-4o's 320ms average, with offline ChatGPT Score 3.99/3.47 (S2TIF/S2SIF), ASR-WER 10.82, UTMOS 3.93, outperforming SpeechGPT (2.98/2.19, WER 45.0) and cascaded SALMONN+Orca / Qwen2-Audio+Orca baselines on InstructS2S-Eval. | Speech decoder trained on only ~1K h, so ASR-WER is higher than industrial-TTS cascades; latency-vs-quality tradeoff at small chunk sizes (lower UTMOS); turn-based rather than full-duplex (no simultaneous listening while speaking). | high |
| yeah2025backchannel | 2025 | turn-taking model (VAP fine-tuning; backchannel prediction) | 109 WOZ attentive-listening sessions (~7-8 min each, Japanese); 13,601 backchannels (5,912.6s); VAP pre-training ~35h | F1 for frame-wise backchannel timing and continuer/assessment type prediction; prosody-flattening sensitivity | Multi-task VAP with pre-training achieves the best F1 (38.11 continuer; 31.76 assessment) versus baselines (e.g., 19.74/14.13), with intensity flattening degrading F1 by up to 3.03 points, showing prosody reliance. | Japanese-only, WoZ attentive-listening setting; low absolute F1 reflects heavily unbalanced real data (~10% positive) | high |
| zhang2025listen | 2025 | LSLM: end-to-end listening-while-speaking model fusing a decoder-only token TTS (106M, 12 blocks) with a streaming vq-wav2vec SSL encoder (34M); early/middle/late fusion compared (middle best); IRQ interruption token stops generation within mu=0.5 s | LibriTTS 585 h TTS pairs + Say_Honey / Speech Commands (51,088 train) interruption data + MUSAN noise; 106M-param AR backbone; 20 epochs | WER of generated speech (Whisper large v3), Precision/Recall/F1 of turn-taking (success = stops within [0,2mu]=1 s of interruption) | LSLM-MF keeps WER 4.05% clean / 4.51% noisy (vs 4.28% vanilla TTS) with interruption-detection F1 98.0% in command-based FDM, degrading to 85.15% F1 and WER 8.50% in voice-based FDM with unseen speakers. | Small 106M model; interruptions are simulated keyword/word commands, not natural barge-ins; no semantic dialogue content beyond TTS utterances; no real-time latency measurement | high |
| zhang2025salmonn | 2025 | SALMONN-omni: standalone codec-free full-duplex speech LLM - Mamba streaming speech encoder (32 Mamba blocks, 2048-dim, 25Hz, distilled from Whisper-large-v3) + single LLM backbone (no codec tokens in vocabulary) + streaming speech synthesizer connected via hidden embeddings; interleaved environment/assistant streams; dynamic 'thinking' mechanism for speak/listen state transitions; three-stage training incl. RL (DPO) for dialogue dynamics | training: LibriSpeech-960h + GigaSpeech-M (~480k samples) ASR, ~730k QA samples (Llama/Web/TriviaQA/SQuAD/NQ/VoiceAssistant-400K/UltraChat), GPT-4o-generated context-dependent barge-in and backchannel data; 80ms time blocks, 320ms generation delay | spoken QA accuracy (Llama Questions, Web Questions, TriviaQA), AlpacaEval GPTScore (Voicebench), turn-taking prediction, barge-in vs backchannel discrimination; full-duplex vs half-duplex modes | SALMONN-omni sets SOTA in full-duplex spoken interaction with an average 35.9% relative improvement over prior open-source full-duplex speech LLMs (and >=30% relative gain claimed), using a codec-free architecture with 320ms delay between listening and speech generation, further improved by DPO for turn-taking/barge-in timing. | Full-duplex mode trades accuracy vs half-duplex; Mamba encoder distilled (not trained on all audio types); 320ms delay still above human turn gaps; RL stage depends on preference data construction. | high |
| zhang2025talking | 2025 | benchmark + user study (turn-taking dynamics; judge model) | ~4h human-AI conversation per system, ~5min sessions, 11 participants; Moshi vs cascaded (Silero VAD+Whisper tiny+SmolLM-135M+Melo TTS); judge trained on Switchboard | corpus-level IPU/gap/overlap statistics + timing-centric judge-model metrics (speak-up, backchannel, interrupt, floor-hold, handle-interrupt) | Moshi leaves only 11.8% cumulative silence gap vs 32.4% for the cascaded system but interrupts too aggressively and rarely backchannels, and unlike the VAD-based system Moshi mostly keeps speaking after user interruptions. | user study participants were authors/colleagues; judge model trained on one dialogue corpus may limit generalization (noted by Full-Duplex-Bench) | high |
| codeclm2025 | 2025 | position/analysis paper on codec-LM co-design | n/a (analysis; RVQ-based audio codecs + codec LMs) | n/a (qualitative argument) | Argues that codec LMs are held back by treating the codec as fixed: neural codecs (RVQ) and the language model should be co-designed so the discrete code space matches LM objectives, rather than adapting LMs to an arbitrary codec. | student workshop position paper; no large-scale experiments in abstract | moderate |

## References

1. Défossez, Alexandre et al. (2024). *Moshi: a speech-text foundation model for real-time dialogue*. arXiv preprint. Full text read. Dual-stream speech-text model with 7B backbone + 1.2B temporal transformer; full-duplex via simultaneous audio+text streams; Mimi codec; first open full-duplex model. canonical dual-stream full-duplex architecture; establishes the multi-stream + inner-monologue recipe [doi:10.48550/arxiv.2410.00037](https://doi.org/10.48550/arxiv.2410.00037)
2. Chaoyou, Fu, et al. (2025). *VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction*. NeurIPS 2025. Full text read. Omni-modal real-time interaction; vision+speech; duplex voice. Evidence for the progressive multi-stage alignment recipe (vision first, audio input, then audio output) as an alternative to native duplex pretraining. [doi:10.52202/085713-2529](https://doi.org/10.52202/085713-2529)
3. Wenyi, Yu, et al. (2025). *SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation*. NeurIPS 2025. Full text read. Codec-free full-duplex: continuous speech encoder+decoder integrated in LLM, no audio tokens. codec-free duplex alternative: hidden-embedding integration + explicit thinking tokens + first RL (DPO) for full-duplex dialogue dynamics [doi:10.52202/085713-0829](https://doi.org/10.52202/085713-0829)
4. Wang, Peng et al. (2024). *A Full-duplex Speech Dialogue Scheme Based On Large Language Models*. NeurIPS 2024. Abstract only. Early LLM full-duplex scheme: simultaneous speech in/out with streaming generation. Demonstrates explicit control tokens plus a lightweight FSM for interrupt/respond decisions - an alternative to implicit end-to-end duplex learning. [doi:10.52202/079017-0427](https://doi.org/10.52202/079017-0427)
5. Hassid, Michael et al. (2023). *Textually Pretrained Speech Language Models*. NeurIPS 2023. Abstract only. TWIST: text-pretrained speech LM; two-stage pretraining recipe foundational for speech LMs. Foundational evidence that text-LLM warm-start beats audio-only pretraining - the standard initialization choice for full-duplex LLM training. [doi:10.52202/075280-2771](https://doi.org/10.52202/075280-2771)
6. Si, Shuzheng et al. (2023). *SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents*. NeurIPS 2023. Full text read. 8k human-human spoken task dialogues; the standard spoken dialogue eval corpus. benchmark quantifying the spoken-vs-text dialogue gap; useful as an evaluation target for speech dialogue training [doi:10.52202/075280-1699](https://doi.org/10.52202/075280-1699)
7. Wang, Qichao et al. (2025). *NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction*. ICML 2025. Full text read. Next-token-pair prediction over dual-channel dialogue (both speakers in one stream); direct full-duplex LM training objective. Shows joint speaker-pair token prediction is a parameter/latency-efficient alternative to dual-stream conditional architectures (Moshi) for learning turn-taking from dual-channel audio. [doi:10.48550/arxiv.2506.00975](https://doi.org/10.48550/arxiv.2506.00975)
8. Wu, Anqi et al. (2025). *Aligning Spoken Dialogue Models from User Interactions*. ICML 2025. Full text read. Preference alignment of spoken dialogue models from real user interactions; RLHF for spoken dialogue. First full pipeline adapting offline preference learning to full-duplex speech, showing timing and content can be jointly aligned from organic user interactions. [doi:10.48550/arxiv.2506.21463](https://doi.org/10.48550/arxiv.2506.21463)
9. Ji, Shengpeng et al. (2025). *WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling*. ICLR 2025. Full text read. Very low-bitrate single-codebook audio tokenizer (40-75 tokens/s) for speech LMs. token-efficiency recipe for duplex LMs: 40-75 tokens/s single-stream codec cuts sequence length (and thus latency) by ~10x vs multi-codebook codecs [doi:10.48550/arxiv.2408.16532](https://doi.org/10.48550/arxiv.2408.16532)
10. Arora, Siddhant et al. (2025). *Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics*. ICLR 2025. Full text read. Turn-taking benchmark for audio foundation models; models poorly predict turn boundaries. Establishes the supervised-judge evaluation protocol for conversational timing, separating corpus-level statistics from event-timing quality. [doi:10.48550/arxiv.2503.01174](https://doi.org/10.48550/arxiv.2503.01174)
11. Fang, Qingkai et al. (2025). *LLaMA-Omni: Seamless Speech Interaction with Large Language Models*. ICLR 2025. Full text read. Speech-in speech-out LLaMA; low-latency streaming; OmniInstruct dataset. low-resource recipe (4 GPUs, 3 days, 200K samples) for low-latency S2S; also documents SpeechGPT's >4500ms latency [doi:10.48550/arxiv.2409.06666](https://doi.org/10.48550/arxiv.2409.06666)
12. Ma, Ziyang et al. (2025). *Language Model Can Listen While Speaking*. AAAI 2025. Full text read. Full-duplex training strategy: keep ASR branch alive while generating speech; two-stream design for listening-while-speaking. Established the channel-fusion design space (early/middle/late) and IRQ-token mechanism that later full-duplex models (Moshi, NTPP) build upon. [doi:10.1609/aaai.v39i23.34665](https://doi.org/10.1609/aaai.v39i23.34665)
13. Ye, Zhen et al. (2025). *Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model*. AAAI 2025. Full text read. Shows codec tokens carry limited semantics; proposes semantically-enriched codec training. evidence that codec token semantics directly drive downstream WER - a key constraint on tokenizer choice for speech generation LLMs [doi:10.1609/aaai.v39i24.34761](https://doi.org/10.1609/aaai.v39i24.34761)
14. Zhang, Qinglin et al. (2025). *OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation*. ACL 2025. Full text read. Flattens audio-text history into one GPT sequence for seamless voice conversation; streaming duplex. flattening + staged half-duplex-to-full-duplex curriculum with simulated barge-in data; best open turn-taking numbers vs Moshi [doi:10.18653/v1/2025.acl-long.709](https://doi.org/10.18653/v1/2025.acl-long.709)
15. Hu, Ke et al. (2025). *Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model*. Interspeech 2025. Full text read. SALM-Duplex: direct duplex modeling without special tokens; interruption/overlap handling in streaming. shows duplex capability can be trained directly from an LLM with zero speech pretraining using turn-level aligned dual channels + barge-in augmented synthetic data [doi:10.21437/interspeech.2025-874](https://doi.org/10.21437/interspeech.2025-874)
16. Ohashi, Atsumoto et al. (2025). *Towards a Japanese Full-duplex Spoken Dialogue System*. Interspeech 2025. Full text read. First full-duplex spoken dialogue system for Japanese; training data construction and evaluation. Second independent language-transfer recipe (tokenizer reinit + stereo fine-tune + synthetic TTS augmentation) confirming that target-language-specific duplex behaviors (overlaps) emerge from data. [doi:10.21437/interspeech.2025-2564](https://doi.org/10.21437/interspeech.2025-2564)
17. Peng, Yizhou et al. (2025). *FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems*. Interspeech 2025. Full text read. Benchmarking pipeline for full-duplex systems: interruption, overlap, latency metrics. Contributes an automated, scalable pipeline for measuring interruption handling (barge-in rates and delays) without human evaluation. [doi:10.21437/interspeech.2025-739](https://doi.org/10.21437/interspeech.2025-739)
18. Hou, Yunfei et al. (2025). *SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant*. Interspeech 2025. Full text read. Speech conversation benchmark incl. turn-taking and interruption abilities. Confirms that for content quality, speech LLMs still lag text backbones and cascades; complements (not a substitute for) duplex-specific benchmarks. [doi:10.21437/interspeech.2025-1950](https://doi.org/10.21437/interspeech.2025-1950)
19. Elmers, Mikey et al. (2025). *Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems*. Interspeech 2025. Abstract only. VAP extended to three-party conversation for turn-taking prediction. Extends the VAP training recipe from dyadic to multi-party turn-taking — relevant for barge-in and side-speech rejection. [doi:10.21437/interspeech.2025-2660](https://doi.org/10.21437/interspeech.2025-2660)
20. Russell, Sam O’Connor & Harte, Naomi (2025). *Prompt-Guided Turn-Taking Prediction*. Interspeech 2025. Abstract only. Prompt-based turn-taking prediction; shows LLMs can do turn-taking with prompting. Documents how sensitive predictive turn-taking models are to acoustics — relevant for training robust turn-taking components. [doi:10.21437/interspeech.2025-668](https://doi.org/10.21437/interspeech.2025-668)
21. Casanova, Edresson et al. (2025). *Open Full-duplex Voice Agent with Speech-to-Speech Language Model*. ASRU 2025. Abstract only. Open full-duplex voice agent; streaming S2S LM with interruption support. Suggests a lightweight conversion recipe (LLM + streaming I/O + serving optimization) exists and is reproducible from open-source code. [doi:10.1109/asru65441.2025.11434669](https://doi.org/10.1109/asru65441.2025.11434669)
22. Lin, Guan-Ting et al. (2025). *Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities*. ASRU 2025. Full text read. Turn-taking-focused full-duplex benchmark: overlap, barge-in, response latency. First scenario-driven full-duplex benchmark: defines the TOR/JSD/latency metric suite later reused by nearly every FD benchmark. [doi:10.1109/asru65441.2025.11433838](https://doi.org/10.1109/asru65441.2025.11433838)
23. Udupa, Sathvik et al. (2025). *Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training*. ASRU 2025. Abstract only. Codec-based streaming endpointer; when to stop listening / start responding. Low-latency endpointing recipe (NAC features + label delay) that can be bolted onto codec-based speech LLMs without retraining the LLM. [doi:10.1109/asru65441.2025.11434752](https://doi.org/10.1109/asru65441.2025.11434752)
24. Jiang, Yiheng & Tian, Biao (2025). *A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions*. ICASSP 2025. Full text read. AEC for mobile full-duplex interaction; the self-talk problem when speaking while listening. Addresses the acoustic precondition of full-duplex (echo-free simultaneous capture) with a deployable small-footprint recipe. [doi:10.1109/icassp49660.2025.10887835](https://doi.org/10.1109/icassp49660.2025.10887835)
25. Li, Guojian et al. (2026). *Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue*. ICASSP 2026. Full text read. Acoustic+linguistic turn-taking for full-duplex; robust overlap handling. Open 1,145h turn-state training corpus and a lightweight bimodal recipe showing acoustic+linguistic fusion beats either modality alone. [doi:10.1109/icassp55912.2026.11463929](https://doi.org/10.1109/icassp55912.2026.11463929)
26. ZhangGaoSheng et al. (2026). *Human-Machine Full-Duplex Dialogue System with Left-Right Brain Division of Labor and Collaboration*. ICASSP 2026. Abstract only. Two-branch architecture (dual-stream) inspired by brain lateralisation for duplex dialogue. Evidence that modular/multi-agent division-of-labor systems remain competitive for FD interaction — a contrast point to end-to-end training. [doi:10.1109/icassp55912.2026.11462331](https://doi.org/10.1109/icassp55912.2026.11462331)
27. Yu, Haoyuan et al. (2026). *Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems*. ICASSP 2026. Full text read. Semi-cascaded full-duplex: units bridge ASR/LLM/TTS; hybrid pipeline vs E2E. Evidence that unit-level audio-grounded MLLM decisions can replace the ASR-to-LLM turn-taking cascade without any full-duplex training, in a plug-and-play manner. [doi:10.1109/icassp55912.2026.11462680](https://doi.org/10.1109/icassp55912.2026.11462680)
28. Roy, Rajarshi et al. (2026). *PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models*. ICASSP 2026. Full text read. Voice/role control in full-duplex models; controllable conversational behavior. Shows hybrid prompt conditioning (role text + voice sample) in duplex SFT yields SOTA interactivity, voice cloning, and instruction adherence — training recipe for controllable full-duplex agents. [doi:10.1109/icassp55912.2026.11463413](https://doi.org/10.1109/icassp55912.2026.11463413)
29. Zhao, Zhixian et al. (2026). *The ICASSP 2026 Humdial Challenge: Benchmarking Human-Like Spoken Dialogue Systems in the LLM Era*. ICASSP 2026. Abstract only. Shared challenge on human-like (incl. duplex) spoken dialogue in the LLM era. Challenge-definition paper pairing emotional intelligence with full-duplex interaction as inseparable human-likeness requirements. [doi:10.1109/icassp55912.2026.11460911](https://doi.org/10.1109/icassp55912.2026.11460911)
30. Lin, Guan-Ting et al. (2026). *Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models*. ICASSP 2026. Abstract only. Overlap-handling evaluation extension of Full-Duplex-Bench. First fully automated benchmark probing model behavior during speech overlap; frames the responsive-vs-floor-holding strategy axis. [doi:10.1109/icassp55912.2026.11463576](https://doi.org/10.1109/icassp55912.2026.11463576)
31. Zhang, Dong et al. (2023). *SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities*. Findings of EMNLP 2023. Abstract only. First speech-in speech-out LLM (speech modality chain); half-duplex precursor. Early blueprint of the three-stage text-LLM-to-speech adaptation pipeline (pretrain, SFT, chain-of-modality) that later duplex systems extend. [doi:10.18653/v1/2023.findings-emnlp.1055](https://doi.org/10.18653/v1/2023.findings-emnlp.1055)
32. Nguyen, Tu Anh et al. (2023). *Generative Spoken Dialogue Language Modeling*. TACL. Abstract only. dGSLM: generative spoken dialogue LM with turn+interruption tokens; precursor to full-duplex LMs. foundational dual-channel (stereo) dialogue modeling; establishes two-channel training data requirement that later work scales via pseudo-stereo and synthetic data [doi:10.1162/tacl_a_00545](https://doi.org/10.1162/tacl_a_00545)
33. Cuervo, Santiago & Marxer, Ricard (2024). *Scaling Properties of Speech Language Models*. EMNLP 2024. Abstract only. Scaling laws for speech LMs (units vs features; text-supervised). Quantifies the steep compute penalty of learning language directly from speech, motivating text-LLM warm-start / hybrid speech-text training for duplex LLMs. [doi:10.18653/v1/2024.emnlp-main.21](https://doi.org/10.18653/v1/2024.emnlp-main.21)
34. Zhang, Xinrong et al. (2024). *Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models*. EMNLP 2024. Abstract only. Duplex models: continuous-time dialogue as synchronous token generation; text-side full-duplex. TDM slicing is a minimal-cost route to convert any LLM into a duplex model; its alternating-slice fine-tuning data recipe is reusable. [doi:10.18653/v1/2024.emnlp-main.644](https://doi.org/10.18653/v1/2024.emnlp-main.644)
35. Veluri, Bandhav et al. (2024). *Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents*. EMNLP 2024. Full text read. Synchronous LLMs: stream user input and model output simultaneously; full-duplex text agents. introduces time-synchronized interleaved dual-stream training + synthetic-data-heavy recipe (212k h synthetic vs 2k h real) [doi:10.18653/v1/2024.emnlp-main.1192](https://doi.org/10.18653/v1/2024.emnlp-main.1192)
36. Park, Se Jin et al. (2024). *Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation*. ACL 2024. Abstract only. Spoken dialogue model with face-to-face cues; turns as dialogue acts. Contributes a 340h parallel audio-visual dialogue corpus and shows text-LLM warm-start with joint speech-text pretraining works for spoken dialogue adaptation. [doi:10.18653/v1/2024.acl-long.860](https://doi.org/10.18653/v1/2024.acl-long.860)
37. Lin, Guan-Ting et al. (2024). *Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations*. ACL 2024. Full text read. Speaking-style-annotated dialogue data and training for spoken LLMs. contributes a style-conditioned spoken dialogue data-generation recipe (LLM rewriting + expressive TTS + human filtering) [doi:10.18653/v1/2024.acl-long.358](https://doi.org/10.18653/v1/2024.acl-long.358)
38. Maimon, Gallil et al. (2025). *Slamming: Training a Speech Language Model on One GPU in a Day*. Findings of ACL 2025. Abstract only. Efficient single-GPU speech LM training recipe (greedy feature sparsification). Shows synthetic-data + preference-optimization recipes can beat scaling-law predictions for SLM training - a cost template for duplex LM training. [doi:10.18653/v1/2025.findings-acl.631](https://doi.org/10.18653/v1/2025.findings-acl.631)
39. Peng, Yifan et al. (2025). *VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Training*. NAACL 2025. Abstract only. Single-stage joint speech-text training for speech-enabled LLMs. Evidence that a single-stage joint speech-text SFT avoids multi-stage pipelines and catastrophic forgetting - a cheap recipe for adding duplex speech to LLMs. [doi:10.18653/v1/2025.naacl-long.298](https://doi.org/10.18653/v1/2025.naacl-long.298)
40. Inoue, Koji et al. (2025). *Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection*. NAACL 2025. Full text read. Backchannel prediction for continuous listening; VAP fine-tuning. Shows a small pre-trained VAP can be fine-tuned into a real-time continuous backchannel predictor — a lightweight alternative to LLM-level duplex training. [doi:10.18653/v1/2025.naacl-long.367](https://doi.org/10.18653/v1/2025.naacl-long.367)
41. Chen, Wenxi et al. (2025). *SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training*. Findings of ACL 2025. Abstract only. Single-stage full-duplex-ish voice interaction with timbre control. Shows single-stage training plus token-sequence compression can reach competitive spoken dialogue performance - evidence against mandatory multi-stage pipelines. [doi:10.18653/v1/2025.findings-acl.115](https://doi.org/10.18653/v1/2025.findings-acl.115)
42. Cui, Wenqian et al. (2025). *Recent Advances in Speech Language Models: A Survey*. ACL 2025. Full text read. Authoritative survey of speech LMs incl. spoken dialogue; taxonomy of architectures. Foundational reference for the training-recipe component choices (tokenizer objectives, stages, alignment) underlying FD models. [doi:10.18653/v1/2025.acl-long.682](https://doi.org/10.18653/v1/2025.acl-long.682)
43. Liu, Wenrui et al. (2025). *Analyzing and Mitigating Inconsistency in Discrete Speech Tokens for Neural Codec Language Models*. ACL 2025. Abstract only. Token inconsistency across codec layers; mitigation for codec LM training. Token-sequence inconsistency is a data-quality hazard for any codec-token duplex LM - relevant when synthesizing or aligning duplex training streams. [doi:10.18653/v1/2025.acl-long.1498](https://doi.org/10.18653/v1/2025.acl-long.1498)
44. Wang, Yuhao et al. (2025). *VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation*. EMNLP 2025. Abstract only. Multi-token prediction cuts speech generation latency — key for duplex responsiveness. MTP is a decoding-side lever to cut full-duplex response latency without quality loss - complements data- and architecture-side training choices. [doi:10.18653/v1/2025.emnlp-main.989](https://doi.org/10.18653/v1/2025.emnlp-main.989)
45. Yan, Ruiqi et al. (2025). *URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models*. Findings of EMNLP 2025. Abstract only. E2E spoken dialogue benchmark: understanding, response, overall quality. Cited as a comprehensive E2E spoken-dialogue evaluation suite covering understanding plus generation. [doi:10.18653/v1/2025.findings-emnlp.933](https://doi.org/10.18653/v1/2025.findings-emnlp.933)
46. Chen, Y.W. et al. (2025). *InteractSpeech: A Speech Dialogue Interaction Corpus for Spoken Dialogue Model*. Findings of EMNLP 2025. Abstract only. Speech dialogue interaction corpus for training spoken dialogue models. Cited as a speech-dialogue interaction corpus for SDM training. [doi:10.18653/v1/2025.findings-emnlp.424](https://doi.org/10.18653/v1/2025.findings-emnlp.424)
47. Chiba, Yuya & Higashinaka, Ryuichiro (2025). *Incorporating Dialogue State Tracking into Japanese Full-duplex Task-oriented Spoken Dialogue Model*. Findings of IJCNLP 2025. Abstract only. DST inside a Japanese full-duplex model; task-oriented duplex dialogue. Shows how to add dialogue management (DST) on top of a full-duplex model — task-oriented capability for duplex systems. [doi:10.18653/v1/2025.findings-ijcnlp.49](https://doi.org/10.18653/v1/2025.findings-ijcnlp.49)
48. Zhang, He et al. (2026). *MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models*. Findings of ACL 2026. Full text read. Multi-round full-duplex evaluation benchmark. Introduces turn segmentation of continuous FD dialogues enabling turn-by-turn multi-round evaluation across quality, instruction following, and safety. [doi:10.18653/v1/2026.findings-acl.263](https://doi.org/10.18653/v1/2026.findings-acl.263)
49. Züfle, Maike et al. (2026). *F-Actor: Controllable Conversational Behaviour in Full-Duplex Models*. Findings of ACL 2026. Full text read. Controllable conversational behaviour (speaking style, overlap behaviour) in full-duplex models. Evidence that controllable duplex behavior (voice, topic, backchannel/interruption counts, dialogue initiation) can be injected via prompt conditioning in a cheap single-stage SFT on synthetic data. [doi:10.18653/v1/2026.findings-acl.242](https://doi.org/10.18653/v1/2026.findings-acl.242)
50. Chen, Yifu et al. (2026). *Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models*. ACL 2026. Full text read. Reward model with semantic + turn-taking axes for aligning duplex dialogue models. Provides the missing low-latency, interpretable reward signal (content vs timing decoupled) needed for RL training of full-duplex LLMs, positioned between VAD-heuristic and offline DPO approaches. [doi:10.18653/v1/2026.acl-long.6](https://doi.org/10.18653/v1/2026.acl-long.6)
51. Jung, Kyudan et al. (2026). *Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models*. ACL 2026 Industry. Abstract only. Industrial multi-turn audio pre-processing pipeline for full-duplex models. Industrial data-pipeline recipe for building full-duplex training corpora from multi-speaker audio. [doi:10.18653/v1/2026.acl-industry.18](https://doi.org/10.18653/v1/2026.acl-industry.18)
52. Wang, Enzhi et al. (2026). *RealTalk-CN: A Realistic Chinese Speech-Text Dialogue Benchmark With Cross-Modal Interaction Analysis*. ACL 2026. Full text read. Chinese spoken dialogue benchmark with cross-modal interaction analysis. Contributes a realistic Chinese speech-text TOD resource with disfluency labels and dynamic modality switching — useful for training robust speech-side understanding. [doi:10.18653/v1/2026.acl-long.131](https://doi.org/10.18653/v1/2026.acl-long.131)
53. Gosai, Advait et al. (2026). *Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction*. ACL 2026. Abstract only. Multi-turn spoken dialogue evaluation on natural interactions. Cited as a multi-turn, natural-interaction spoken dialogue evaluation benchmark. [doi:10.18653/v1/2026.acl-long.1654](https://doi.org/10.18653/v1/2026.acl-long.1654)
54. Lin, Guan-Ting et al. (2026). *Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner*. ACL 2026 (short). Abstract only. Multi-turn duplex evaluation with automated examiner. Cited as the first dynamic multi-turn FD evaluation with an automated examiner. [doi:10.18653/v1/2026.acl-short.4](https://doi.org/10.18653/v1/2026.acl-short.4)
55. Chung, Ho-Lam et al. (2026). *LLM-Codec: Neural Audio Codec Meets Language Model Objectives*. Findings of ACL 2026. Full text read. Codec trained with LM objectives; codec-LM co-training for speech LMs. closes the loop on 'codec does matter': co-training the codec with LM objectives makes tokens easier for speech LMs to predict [doi:10.18653/v1/2026.findings-acl.1308](https://doi.org/10.18653/v1/2026.findings-acl.1308)
56. Chen, Yiming et al. (2026). *VoiceBench: Benchmarking LLM-Based Voice Assistants*. TACL. Abstract only. Voice assistant benchmark covering ASR-robustness, knowledge, instruction following. Grounds the robustness-variation axis (accents, noise, mispronunciation) that full-duplex benchmarks later inherit. [doi:10.1162/tacl.a.628](https://doi.org/10.1162/tacl.a.628)
57. Lu, Xiangang et al. (2025). *DuplexMamba: Enhancing Real-Time Speech Conversations with Duplex and Streaming Capabilities*. PRICAI 2025 (LNCS). Full text read. Mamba-based duplex speech conversation with streaming; linear-time architecture. Demonstrates a linear-complexity SSM backbone gives constant-memory streaming duplex and that explicit state-token supervision (interrupt vs ignore discrimination) is an effective training signal. [doi:10.1007/978-981-95-3346-6_5](https://doi.org/10.1007/978-981-95-3346-6_5)
58. Aylett, Matthew P. et al. (2023). *Why is my Agent so Slow? Deploying Human-Like Conversational Turn-Taking*. ICMI 2023. Abstract only. Latency and turn-taking in spoken agents; user-perceived slowness of pipelined systems. motivational evidence that turn-taking latency is a UX problem full-duplex training aims to solve [doi:10.1145/3623809.3623976](https://doi.org/10.1145/3623809.3623976)
59. Kendrick, Kobin H. et al. (2023). *Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions*. Philosophical Transactions of the Royal Society B. Abstract only. Evidence that human turn-taking is multimodal and overlapping — motivation for duplex agents. human-behavior evidence that turn-taking cues are multimodal - supports multimodal (not VAD-only) turn prediction in duplex agents [doi:10.1098/rstb.2021.0473](https://doi.org/10.1098/rstb.2021.0473)
60. Patamia, Rutherford Agbeshi et al. (2025). *Turn-Taking Modelling in Conversational Systems: A Review of Recent Advances*. Technologies. Abstract only. Review of turn-taking models for conversational systems. conceptual vocabulary (TFO, IPU, hold/shift) for evaluating turn-taking quality of duplex LLMs [doi:10.3390/technologies13120591](https://doi.org/10.3390/technologies13120591)
61. Xie, Zhifei & Wu, Changqiao (2024). *Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming*. arXiv preprint. Full text read. Open full-duplex speech model; streaming audio in/out; text-guided training; batch-offline RL for duplex. shows 0.5B-scale recipe: text-instructed parallel audio decoding + small synthetic speech QA data [doi:10.48550/arxiv.2408.16725](https://doi.org/10.48550/arxiv.2408.16725)
62. Zhifei, Xie, & Wu, Chyuan-Chuan (2024). *Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities*. arXiv preprint. Full text read. Multi-modal (vision+speech) duplex; streaming modalities. introduces learned interruption tokens (irq/n-irq) as an alternative to true duplex modeling [doi:10.48550/arxiv.2410.11190](https://doi.org/10.48550/arxiv.2410.11190)
63. Wang, Xiong et al. (2024). *Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM*. arXiv preprint. Full text read. Frozen text LLM + lightweight speech in/out modules; low-latency S2S dialogue. evidence that a frozen LLM + adapters can reach spoken QA parity while avoiding catastrophic forgetting; VAD-based chunk state prediction for barge-in [doi:10.48550/arxiv.2411.00774](https://doi.org/10.48550/arxiv.2411.00774)
64. Xu, Jin et al. (2025). *Qwen2.5-Omni Technical Report*. arXiv preprint. Abstract only. Thinker-Talker architecture; unified audio-text tokens; full-duplex streaming; AnyTalk. Origin of Thinker-Talker; shows concurrent text+speech generation via hidden-state reuse, a foundational FD-relevant architectural pattern. [doi:10.48550/arxiv.2503.20215](https://doi.org/10.48550/arxiv.2503.20215)
65. Jin, Xu et al. (2025). *Qwen3-Omni Technical Report*. arXiv preprint. Full text read. Next-gen omni model; faster duplex interaction. Key evidence that mixing unimodal+cross-modal data in early text pretraining avoids modality degradation — a training-recipe data point for FD omni models. [doi:10.48550/arxiv.2509.17765](https://doi.org/10.48550/arxiv.2509.17765)
66. Qian, Chen, et al. (2025). *MinMo: A Multimodal Large Language Model for Seamless Voice Interaction*. arXiv preprint. Full text read. Alibaba full-duplex voice LLM; 8.8M hours speech data; MinMo-800B. Canonical example of bolting a semantic turn-taking predictor onto an aligned half-duplex model — shows the 'aligned + FD predictor' training path with concrete latency budget. [doi:10.48550/arxiv.2501.06282](https://doi.org/10.48550/arxiv.2501.06282)
67. Zeng, Aohan et al. (2024). *GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot*. arXiv preprint. Abstract only. E2E spoken chatbot with dual-token (text+speech) generation; GLM-4-voice. ultra-low-bitrate single-codebook tokenization (12.5Hz, 175bps) + text-to-token data synthesis at 1T-token scale [doi:10.48550/arxiv.2412.02612](https://doi.org/10.48550/arxiv.2412.02612)
68. Fu, Chaoyou et al. (2024). *VITA: Towards Open-Source Interactive Omni Multimodal LLM*. arXiv preprint. Full text read. Open omni-modal LLM with audio+vision; duplex-capable interaction. system-level duplex recipe (twin-model generation/monitoring + state tokens) - an alternative to in-model duplex [doi:10.48550/arxiv.2408.05211](https://doi.org/10.48550/arxiv.2408.05211)
69. Li, Yadong et al. (2025). *Baichuan-Omni-1.5 Technical Report*. arXiv preprint. Full text read. Full-duplex omni model; 100k-hour dialogue data; incremental decoding. Shows interleaved text+audio-token generation from an LLM plus a compact 12.5 Hz RVQ tokenizer as a practical FD-capable training design. [doi:10.48550/arxiv.2501.15368](https://doi.org/10.48550/arxiv.2501.15368)
70. Junbo, Cui, et al. (2026). *MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction*. arXiv preprint. Full text read. Real-time full-duplex omni-modal model; on-device duplex. Strongest evidence for the 'single-stream time-axis + cheap speech decoder + RL' recipe: full-duplex without generating speech tokens in the main LLM. [doi:10.48550/arxiv.2604.27393](https://doi.org/10.48550/arxiv.2604.27393)
71. Fang, Qingkai et al. (2026). *BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM*. arXiv preprint. Full text read. Native full-duplex in a single autoregressive LLM (no dual streams). Directly answers 'how to train FD': a small structured fine-tune + DPO can convert a turn-based SpeechLM into native full-duplex with no architectural change — key review evidence. [doi:10.48550/arxiv.2606.14528](https://doi.org/10.48550/arxiv.2606.14528)
72. Li, Yadong et al. (2026). *UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction*. arXiv preprint. Full text read. Unified audio front-end handling user speech + self-speech (barge-in aware). Evidence that front-end perception (VAD+TD+interruption detection) can be folded into a single LLM objective rather than cascaded modules — a tractable FD training recipe. [doi:10.48550/arxiv.2604.19221](https://doi.org/10.48550/arxiv.2604.19221)
73. Yan, Ruiqi et al. (2026). *SoulX-Duplug: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation*. arXiv preprint. Full text read. Plug-and-play streaming state prediction for duplex; conversational state machine. Shows the 'semantic VAD via joint streaming ASR' recipe — a lighter-weight alternative to training the full duplex model end-to-end. [doi:10.48550/arxiv.2603.14877](https://doi.org/10.48550/arxiv.2603.14877)
74. Zhou, Dingkun et al. (2026). *S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling*. arXiv preprint. Full text read. Causal streaming reasoning for conversational behaviour in duplex models. Contributes a training-data+annotation methodology: causally-annotated overlapping dialogue data and a hierarchy of behavior states for FD models. [doi:10.48550/arxiv.2602.11065](https://doi.org/10.48550/arxiv.2602.11065)
75. Zhang, Luoyuan et al. (2026). *Liberating LLM Capabilities in Full-Duplex Speech Models*. arXiv preprint. Full text read. Recovers LLM reasoning capabilities degraded by speech training; capability-preserving training. Evidence for a token-schema-only recipe that makes text a first-class output during full-duplex interaction, improving reasoning without architectural change. [doi:10.48550/arxiv.2606.07547](https://doi.org/10.48550/arxiv.2606.07547)
76. Ohashi, Atsumoto et al. (2026). *Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models*. arXiv preprint. Full text read. Aligning full-duplex models on multiple interaction facets (interruption, overlap, backchannel). Central evidence on the RL recipe for FD training: which rewards matter, the semantic-degradation failure mode, and cross-model generalization. [doi:10.48550/arxiv.2606.11167](https://doi.org/10.48550/arxiv.2606.11167)
77. Donghang, Wu, et al. (2026). *The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning*. arXiv preprint. Full text read. Latent 'thinking' tokens between listening and speaking in duplex dialogue. Evidence for using the listening window productively (latent reasoning) instead of predicting <SIL> padding tokens — a training-objective recipe for FD models. [doi:10.48550/arxiv.2603.17837](https://doi.org/10.48550/arxiv.2603.17837)
78. Riera, Pablo et al. (2026). *Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models*. arXiv preprint. Full text read. Synchronization between streams; turn-taking dynamics in duplex models. Provides an analysis methodology (CKA/probing) for evaluating internal turn-taking readiness of FD models — useful for review discussion of FD evaluation. [doi:10.48550/arxiv.2605.20356](https://doi.org/10.48550/arxiv.2605.20356)
79. Lu, Hui et al. (2026). *How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue*. arXiv preprint. Full text read. User-stream routing (how the listening stream feeds the model while it speaks). Frames user-stream routing as a core FD design axis — directly relevant to how to structure input streams when training FD LLMs. [doi:10.48550/arxiv.2605.10199](https://doi.org/10.48550/arxiv.2605.10199)
80. Li, Yuxin et al. (2026). *Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning*. arXiv preprint. Full text read. RL to decouple conversational dynamics (turn-taking vs content) in duplex models. Strongest evidence that the intelligence-dynamics trade-off is not fundamental: timing-policy RL on top of an instruction-tuned base yields SOTA latency and dynamics without sacrificing reasoning. [doi:10.48550/arxiv.2607.07148](https://doi.org/10.48550/arxiv.2607.07148)
81. Hsiao, Chi-Yuan et al. (2026). *ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models*. arXiv preprint. Full text read. RL action-space projection to optimize interactivity in duplex speech LMs. Diagnoses reward hacking in raw-token duplex RL (generative collapse) and shows projecting the action space to binary timing states stabilizes GRPO while preserving semantics. [doi:10.48550/arxiv.2604.10065](https://doi.org/10.48550/arxiv.2604.10065)
82. Pan, Shidong et al. (2025). *Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech*. arXiv preprint. Full text read. Reasoning over conversational behavior (turn-taking decisions) in duplex speech models. Proposes next-behavior reasoning (intent-to-action causal chains) as a complementary layer and evaluation paradigm for full-duplex training beyond token-level prediction. [doi:10.48550/arxiv.2512.21706](https://doi.org/10.48550/arxiv.2512.21706)
83. Wu, Donghang et al. (2025). *Chronological Thinking in Full-Duplex Spoken Dialogue Language Models*. arXiv preprint. Full text read. Time/chronology awareness in duplex dialogue; interruption-consistent state updates. Replaces idle silence-token prediction with thinking-while-listening: streaming reasoning amortized inside the listening window, improving response quality at zero latency cost. [doi:10.48550/arxiv.2510.05150](https://doi.org/10.48550/arxiv.2510.05150)
84. Arora, Siddhant et al. (2025). *Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems*. arXiv preprint. Full text read. CoT in streaming full-duplex E2E spoken dialogue systems. Shows injecting streaming ASR+response text as intermediate CoT targets repairs the semantic-coherence weakness of duplex E2E while retaining overlap capability and real-time latency. [doi:10.48550/arxiv.2510.02066](https://doi.org/10.48550/arxiv.2510.02066)
85. Zhang, Hao et al. (2025). *LLM-Enhanced Dialogue Management for Full-Duplex Spoken Dialogue Systems*. arXiv preprint. Full text read. LLM-driven dialogue management inside full-duplex systems. Advocates lightweight LLM-based dialogue management decoupled from the generator — cheap semantic turn-taking and barge-in classification without retraining the CDE. [doi:10.48550/arxiv.2502.14145](https://doi.org/10.48550/arxiv.2502.14145)
86. Borui, Liao, et al. (2025). *FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems*. arXiv preprint. Full text read. Pluggable full-duplex layer for existing half-duplex systems (interruption, streaming). Explicit Idle state (semantic-integrity buffering) prevents backchannels/noise from reaching the LLM — a plug-and-play alternative to integrated duplex architectures like MinMo/Freeze-Omni. [doi:10.48550/arxiv.2502.13472](https://doi.org/10.48550/arxiv.2502.13472)
87. Singh, Bhaskar et al. (2026). *Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations*. arXiv preprint. Full text read. Full-duplex framework trained on real-world Hindi conversations. Recipe transfer (tokenizer swap + two-stage training on real stereo data) achieves a working full-duplex model for a new language with 26K h where Moshi needed 7M h — data quality over quantity. [doi:10.48550/arxiv.2604.23295](https://doi.org/10.48550/arxiv.2604.23295)
88. Liu, Zhanxun et al. (2025). *X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System*. arXiv preprint. Full text read. Argues modular (cascaded) S2S can match E2E with far less data — the modular vs E2E debate. Engineering counterpoint: a systematically optimized cascade reaches sub-second full-duplex latency with modular flexibility, questioning the necessity of E2E duplex training. [doi:10.48550/arxiv.2512.18706](https://doi.org/10.48550/arxiv.2512.18706)
89. Rubenstein, Paul K. et al. (2023). *AudioPaLM: A Large Language Model That Can Speak and Listen*. arXiv preprint. Full text read. Early speak-and-listen LM (PaLM + audio tokens); half-duplex but foundational. establishes joint speech-text vocabulary + text-LLM initialization recipe that later spoken dialogue models build on [doi:10.48550/arxiv.2306.12925](https://doi.org/10.48550/arxiv.2306.12925)
90. Gong, Yuan et al. (2023). *Listen, Think, and Understand*. arXiv preprint. Full text read. Audio LLM: listening beyond labels, thinking, scene understanding — understanding side of duplex. large-scale audio-QA data recipe (5M tuples) and LoRA-based audio-in/text-out alignment for LLMs [doi:10.48550/arxiv.2305.10790](https://doi.org/10.48550/arxiv.2305.10790)
91. Tang, Changli et al. (2023). *SALMONN: Towards Generic Hearing Abilities for Large Language Models*. arXiv preprint. Full text read. Speech-audio-language understanding; codec-free; hearing abilities. parameter-efficient audio-in/text-out alignment recipe (Q-Former + LoRA, 0.24% trainable) reused by later omni models [doi:10.48550/arxiv.2310.13289](https://doi.org/10.48550/arxiv.2310.13289)
92. Chu, Yunfei et al. (2024). *Qwen2-Audio Technical Report*. arXiv preprint. Full text read. Audio understanding LLM (input side for many duplex systems). recipe for audio-in/text-out instruction following (natural-language prompt pretraining + DPO) used as a component in cascaded voice systems [doi:10.48550/arxiv.2407.10759](https://doi.org/10.48550/arxiv.2407.10759)
93. Zhao, Shuaijiang et al. (2024). *Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data*. arXiv preprint. Abstract only. 60k+ hours synthetic speech dialogue SFT (KE-Omni); data scaling result. extreme SFT-data-scaling recipe (60k+ h synthetic) for speech interaction models [doi:10.48550/arxiv.2412.01078](https://doi.org/10.48550/arxiv.2412.01078)
94. Fu, Yu-Kuan et al. (2024). *Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Speech Generation*. arXiv preprint. Full text read. Pseudo-stereo (two-speaker) data for dialogue speech generation. data recipe for scaling stereo dialogue data from mono sources - directly relevant to training-data scarcity for duplex models [doi:10.48550/arxiv.2407.01911](https://doi.org/10.48550/arxiv.2407.01911)
95. Wang, Minghan et al. (2025). *SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development*. arXiv preprint. Full text read. Automated speech dialogue data generation pipeline for speech-LLM training. Cheap, human-comparable dialogue synthesis pipeline — but reviewers should note it lacks true overlapping-speech structure for FD training. [doi:10.48550/arxiv.2503.23848](https://doi.org/10.48550/arxiv.2503.23848)
96. Nakata, Wataru et al. (2026). *DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Models*. arXiv preprint. Full text read. Scale full-duplex dialogue speech data with speaker separation. Solves the FD training-data bottleneck by converting web-scale monaural podcasts into two-stream speaker-separated dialogue. [doi:10.48550/arxiv.2607.04941](https://doi.org/10.48550/arxiv.2607.04941)
97. Nakata, Wataru et al. (2026). *DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio*. arXiv preprint. Full text read. Recovers duplex dialogue tracks from in-the-wild audio — data construction. Enables reuse of degraded in-the-wild monaural audio as clean two-stream FD training data. [doi:10.48550/arxiv.2604.09344](https://doi.org/10.48550/arxiv.2604.09344)
98. Ge, Yuan et al. (2025). *FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction*. arXiv preprint. Full text read. Full-duplex human-LLM speech interaction benchmark. Adds the model-interrupts-user emergency scenario and argues next token-pair prediction is the path to true duplex modeling. [doi:10.48550/arxiv.2509.22243](https://doi.org/10.48550/arxiv.2509.22243)
99. Ray, Soham et al. (2026). *Tau-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains*. arXiv preprint. Full text read. Real-world-domain benchmark for full-duplex voice agents. Shows full-duplex interaction skill (not just ASR/TTS) is the binding constraint on voice-agent task performance — a quantitative voice-vs-text gap. [doi:10.48550/arxiv.2603.13686](https://doi.org/10.48550/arxiv.2603.13686)
100. Fukuda, Ryo et al. (2026). *M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models*. arXiv preprint. Full text read. Multi-turn, multilingual, multidomain duplex benchmark. Formalizes the single-turn/user-only/full-context evaluation protocol that isolates dialogue-history effects on FD behavior. [doi:10.48550/arxiv.2607.29125](https://doi.org/10.48550/arxiv.2607.29125)
101. Wang, Chengyou et al. (2026). *Full-Duplex Interaction in Spoken Dialogue Systems: A Comprehensive Study from the ICASSP 2026 HumDial Challenge*. arXiv preprint. Full text read. Study of full-duplex interaction from the HumDial challenge submissions. Largest real dual-channel FD evaluation resource; its analysis maps winning designs (cascaded vs semi-cascaded vs E2E; heuristic vs specialized-model vs LLM-judge turn strategies). [doi:10.48550/arxiv.2604.21406](https://doi.org/10.48550/arxiv.2604.21406)
102. Yang, Guangzhao et al. (2026). *JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue*. arXiv preprint. Full text read. Joint acoustic-linguistic turn-taking detection for duplex. Shows joint acoustic-linguistic modeling with shared frozen ASR encoder gives LLM-competitive turn-taking at a fraction of the latency/overhead. [doi:10.48550/arxiv.2603.26515](https://doi.org/10.48550/arxiv.2603.26515)
103. Kangxiang, Xia, et al. (2026). *Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model*. arXiv preprint. Full text read. Interruption detection benchmark/metric/model; semantic awareness. Provides the first real-dialogue interruption benchmark plus an APT metric that prices the responsiveness-robustness trade-off — directly usable to tune barge-in training data. [doi:10.48550/arxiv.2603.24144](https://doi.org/10.48550/arxiv.2603.24144)
104. Chen, Yuxuan & Yu, Hua‐Zhong (2025). *From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models*. arXiv preprint. Full text read. Survey of full-duplex spoken LMs; taxonomy from turn-taking to synchronous dialogue. Provides the conceptual frame (joint vs conditional probability objectives, NTPP) for how to train full-duplex models. [doi:10.48550/arxiv.2509.14515](https://doi.org/10.48550/arxiv.2509.14515)
105. Lu, J et al. (2026). *A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision Space*. arXiv preprint. Full text read. Survey of full-duplex systems: architecture hierarchy, interaction ontology, decision space. Explains WHY training data coverage (not architecture alone) is the binding constraint for full-duplex behavior — central for a how-to-train review. [doi:10.48550/arxiv.2606.19453](https://doi.org/10.48550/arxiv.2606.19453)
106. Ji, Shengpeng et al. (2024). *WavChat: A Survey of Spoken Dialogue Models*. arXiv preprint. Full text read. Survey of spoken dialogue models (incl. duplex approaches). taxonomy + technology overview for the review's framing of duplex training (representations, alignment stages, RL) [doi:10.48550/arxiv.2411.13577](https://doi.org/10.48550/arxiv.2411.13577)
107. Keyu, An, et al. (2024). *FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs*. arXiv preprint. Full text read. SenseVoice (understanding) + CosyVoice (generation) foundation models; ecosystem for spoken LLMs. open ASR/TTS components + semantic tokenizer widely reused to synthesize speech dialogue training data (e.g., OmniFlatten uses CosyVoice tokenizer) [doi:10.48550/arxiv.2407.04051](https://doi.org/10.48550/arxiv.2407.04051)
108. OpenAI et al. (2024). *GPT-4o System Card*. OpenAI (arXiv). Full text read. Commercial reference: GPT-4o real-time speech-to-speech (input=output = text tokens); not peer-reviewed. industry reference point: 232-320ms audio latency is the target that open full-duplex models benchmark against [doi:10.48550/arxiv.2410.21276](https://doi.org/10.48550/arxiv.2410.21276)
109. Modi, Smit Nautambhai et al. (2026). *EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions*. arXiv preprint. Full text read. Benchmark for state-update reasoning when interrupted mid-response — duplex cognition. Supplies an evaluation harness plus a failure taxonomy showing SOTA full-duplex models fail mid-generation state revision, defining concrete training targets (interruption-aware SFT/RL) for duplex LLMs. [doi:10.48550/arxiv.2604.16456](https://doi.org/10.48550/arxiv.2604.16456)
110. Zhong, Tao et al. (2026). *IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems*. arXiv preprint. Full text read. Adaptive fusion against self-talk/noise interference in E2E full-duplex systems. Shows interfering-speaker leakage corrupts duplex LLM conditioning and that lightweight frame-level gating restores turn-taking robustness - a cheap, streaming-safe training/architecture fix. [doi:10.48550/arxiv.2606.06559](https://doi.org/10.48550/arxiv.2606.06559)
111. Wu, Shih-Lun et al. (2025). *Towards Codec-LM Co-design for Neural Codec Language Models*. NAACL 2025 Student Research Workshop. Abstract only. Argues codec and LM should be co-designed for codec language models — tokenizer layer evidence. [doi:10.18653/v1/2025.naacl-srw.6](https://doi.org/10.18653/v1/2025.naacl-srw.6)