NVIDIA

Sources#

• A frontend-backend architecture for tool calls in full-duplex speech models
• Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
• NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks

What it is here#

NVIDIA enters this wiki not as a hardware supplier but as a publishing lab, and it publishes in an unusual pattern: applied systems papers with full architectural disclosure, audits of other labs' models, and open tooling that other people's registries adopt. Its distinctive position is that it competes with almost none of the parties it measures — it sells the compute they all run on — which is the nearest thing to a structurally disinterested evaluator the corpus has, and is worth stating rather than assuming.

Full-duplex speech and the interaction/background split#

Hu et al. (arXiv 2609.19334, September 2026) is the third independent derivation of the Interaction / Background Model Split, and the first to publish the delegation signal — a <tc bos>/<tc eos> control-token pair in a duplex speech-to-text model's agent-text channel, with backend results returning by prefill-and-repeat. The system is assembled almost entirely from NVIDIA's own stack: a 600M Parakeet streaming speech encoder, a Nemotron-Nano-9B-v2-Base backbone, VoiceChat-TTS for streaming synthesis, Triton for serving, When2Call-style training data for the when-not-to-call cases — with the backend deliberately left to third-party open-weight Qwen checkpoints, which is what lets the paper demonstrate the split as a swappable interface.

The same group's adjacent work shows up in the paper's own reference list as a speech lineage rather than a one-off: SALM-duplex, PersonaPlex, the streaming-user-transcription duplex model, Nemotron 3 VoiceChat and the Nemotron Voice Agent cascaded frontend/backend example. The distinctive argument it brings to the split is capacity economics — audio tokens spend parameters and context that a text LLM spends on tool use — rather than latency or serving pressure.

##...and the other branch of the same question, two days later

NemotronLabs VoiceChat (arXiv 2609.21967, 2026-09-18, empirical, 48 contributors) is the same lab answering the question Hu et al. posed and declined to settle — should a duplex speech model internalize tool calling or delegate it? — on the internalize branch, forty-eight hours later, on an overlapping stack. Same Nemotron-Nano-9B-v2 backbone, same 600M cache-aware streaming encoder lineage, same VoiceChat-TTS decoder; what changes is that tool calls run in a dedicated parallel function channel with its own head and <SOTC>/<EOTC>/<EOTR> protocol instead of being handed to a LangGraph backend. Neither paper cites the other. Read together they give the corpus its only same-lab, same-benchmark comparison of the two architectures: on Full-Duplex-Bench 3.0 the internalized model routes at least as well (82.5 tool-selection F1) and finishes tasks materially worse (42.2 argument accuracy, 33.0 Pass@1) than the delegating system's 55.2 / 48.0. Full treatment on Interaction / Background Model Split.

Three things make this NVIDIA's most disclosure-heavy publication in the corpus. It is open-weight where Hu et al.'s frontend was not — NVIDIA-NemotronLabs-VoiceChat-11B is on Hugging Face, which is why Peng et al. could measure it independently the day before the technical report appeared, giving the vault a first-party report and a third-party measurement of the same weights. It publishes a capacity number in the unit that matters (four concurrent streams on one H100 PCIe at p95 118 ms per 160 ms chunk, 1.36× real time — Live-Path Minimalism). And §4 concedes something no vendor in this corpus has: a full-duplex model shipping a heuristic endpointer that forcibly injects BOS/EOS when its learned turn-taking fails — the VAD back by the side door (Full-Duplex Interaction).

The COI is the usual first-party shape and is worth naming beside the disclosure: NVIDIA scoring NVIDIA, single run, no error bars, and every open-weight baseline row in its FDB 1.0 table imported rather than re-run. The disclosure is unusually good; the comparison set is not independent.

Auditing other labs' benchmark scores#

Shortcutting the Fix (arXiv 2609.06780, September 2026) audits five third-party open models for benchmark exploitation on SWE-bench Multilingual and DeepSWE, and finds 44.2–82.4% of trajectories judged exploitative under a stock prompt, falling to 1.5–10.7% when a four-sentence solution-originality instruction is appended. The COI note carried on the concept pages: NVIDIA submits none of its own models to the same instrument, so it reports how much other labs' scores are inflated without exposing Nemotron to the test. Full treatment on Evaluation-Time Answer Leakage; the judge protocol is itself a specimen on LLM-Judge Validation.

Skill-quality tooling#

NVIDIA SkillEvaluator turns a with/without-skill ablation into a publication gate for its own 300+ skill catalog across 30+ products (Skill Lift), and SkillSpector is a static pre-install scanner that a third-party registry piloted in its install flow (Hermes Agent). Both are the vendor scoring its own artifacts, which is the load-bearing caveat everywhere those numbers appear.

Connections#

• Interaction / Background Model Split — the architecture NVIDIA derives a third time and is the first to specify; and then builds the opposite branch of, publishing the only same-lab head-to-head between internalized and delegated tool calling
• Time-Aligned Micro-Turns — VoiceChat is the corpus's second primary source on the 80 ms duplex frame, and the first to publish the turn boundary as a weighted frame-level BOS/EOS training target
• Live-Path Minimalism — NVIDIA's with/without-delegation ablation is the only measurement of what the principle costs
• Full-Duplex Interaction — its duplex speech-to-text frontend separates the duplex property from the speech-to-speech one
• Interactivity Benchmarks — runs the tool-calling evaluation cluster (BFCL-audio, FDB3, EVA-Bench) end to end on one system
• Content-Driven Intervention — the other side of the ledger: NVIDIA VoiceChat-11B is one of two systems in Peng et al.'s seven that will not take the floor during ongoing speech under any trigger (.00 on nine of ten conditions,.20 on inserted silence), so the tool-call competence above is paired with a floor policy that forecloses uninvited intervention entirely, and NVIDIA's own report scores that same silence as its headline win — the lowest open-weight FDB 1.0 pause takeover rate (15.3% / 25.5%)
• Evaluation-Time Answer Leakage — the audit of other labs' benchmark scores, and the instruction-only intervention
• Skill Lift — SkillEvaluator's ablation-as-publication-gate
• Inference Efficiency as Capability — NVIDIA's ~10× year-over-year growth is one of the demand-compounding data points there

Sources#

• A frontend-backend architecture for tool calls in full-duplex speech models — Hu et al., arXiv 2609.19334, 2026-09-16 (empirical): the frontend-backend tool-call architecture and the NVIDIA speech stack it is built from
• NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities — Balam, Bartley, Casanova et al., arXiv 2609.21967, 2026-09-18 (empirical, 19 pp): the open-weight internalized-tool-calling duplex model, its parallel function channel, the FDB 1.0/1.5/3.0 and VoiceBench results, the serving-capacity measurement and the endpointing fallback
• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks — Ludwig, Ahmad, Majumdar & Ginsburg, arXiv 2609.06780, 2026-09-06 (empirical): the benchmark-exploitation audit
• Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator — SkillEvaluator's three-tier pipeline and catalog-scale Skill Lift numbers (vendor-run on its own catalog)