H
Howardism
Howardism · Vol. 03Plate II · No. 02

Interaction & Multimodal, in order.

Notes9DomainInteraction & MultimodalOpen Qs11Newest4 Aug 2026Oldest13 May 2026

Real-time, multimodal, full-duplex, and human-AI collaboration.

Map of Content for the interaction-multimodal domain — 9 concepts. Curated entry point; see Home for all domains.

  • Encoder-Free Early Fusion — Multimodal design with minimal pre-processing instead of large standalone encoders: TML co-trains dMel audio + 40×40-patch hMLP + flow head in one transformer for 200ms latency; Gemma 4's 12B independently discards a 305M audio conformer for on-device memory; Inkling carries the design to 975B open-weight scale — but Kimi K3 keeps a 401M MoonViT-V2 encoder at 2.8T and tops the corpus on exactly the dense-text-in-image tasks where the encoder-free 12B regressed, so the verdict is now contested
  • Full-Duplex Interaction — Perceive-and-respond simultaneously across modalities; proactive interjection, visual-cue reactions, simultaneous speech, live translation/commentary, time-aware speech — all special cases of model behavior; in production audio-only form since July 2026 as GPT-Live's voice model
  • Interaction / Background Model Split — Dual-model architecture: time-aware interaction model stays present; async background model handles deep reasoning/tools; rich-context-package delegation; "reasoning-model planning at non-thinking latency"; Inkling (July 2026) is the named background half — and OpenAI's GPT-Live (July 2026) ships the same split in production, delegating from a full-duplex voice model to GPT-5.5 over a pre-warmed prefilled session
  • Interaction Models — Thinking Machines Lab (May 2026): models that handle audio/video/text interaction natively in real time instead of via harness; interactivity scales with intelligence only if it's in the model — with OpenAI's GPT-Live (July 2026) independently shipping the audio slice of the same conclusions in production
  • Interactivity Benchmarks — FD-bench, Audio MultiChallenge + new TimeSpeak/CueSpeak (proactive audio) and RepCount-A/ProactiveVideoQA/Charades (visual proactivity); TML-Interaction-Small: 0.40s turn-taking latency, dominates interaction quality
  • Live-Path Minimalism — GPT-Live's serving principle — "the voice must flow": the realtime media loop is the only thing on the live path; delegation, context compaction, persistence, and instance management all run asynchronously off it. Stateful-instance handoff (warm a replacement, prefill, run both in parallel, cut over) turns compaction and rebalancing into zero-interruption transitions; delegation is a budgeted loop over a pre-warmed prefilled frontier-model session; WARP + Instant Connect collapse WebRTC startup from six round trips to a single UDP packet; capacity is concurrent sessions keeping every frame on schedule, not GPU throughput
  • Time-Aligned Micro-Turns — The core interaction-model move: input/output as continuous streams in ~200ms interleaved chunks, no turn boundaries; streaming-sessions inference (upstreamed to SGLang), latency-tuned MoE kernels, bitwise trainer-sampler alignment
  • Turn-Based Interface Bottleneck — Why current AI interfaces limit collaboration: single-thread turn-taking is a bandwidth bottleneck; humans pushed out by the interface, not the work; less-intelligent harness (VAD/turn-detection) should dissolve — and did: GPT-Live removed the turn detector from the production audio path (July 2026), with turns surviving only as a derived application-layer view
  • Why AI Lags at Design — Andrew Ambrosino's four reasons frontier models are worse at visual/product design than at code: design is hard to grade (no clean reward like 'does it compile'), it sat outside the AI-research flywheel labs optimized for, it rewards novelty where code rewards known patterns, and it hides a design↔code abstraction layer (a rebrand is 263 components on the surface, semantic relationships underneath)

Open questions 11 open

  • Encoder-Free Early Fusion3 open
    • SourceDoes an encoder-free model at matched size still match? Neither source runs the ablation. TML co-trains everything from scratch; Gemma 4 freezes encoders on four models and drops them on one, at a different scale. Kimi K3 retains a 401M encoder at 2.8T and leads on dense-text vision, which sharpens the question without answering it — still no matched-size arm anywhere.
    • SourceIs the dense-text degradation intrinsic to a projection-only vision path, or an artifact of the 12B's particular training run? The prediction is falsifiable: an encoder-free 31B should show the same InfographicVQA cliff at 280 tokens.
    • SourceTML deletes encoders and co-trains from scratch. Gemma 4 deletes encoders and trains the 12B from scratch, but keeps frozen encoders elsewhere. Which half of "encoder-free + from-scratch" does the work?
  • Interaction Models3 open
    • WaitDoes the interaction/background split generalize, or is it a transitional artifact until a single model is both fast and deep enough? Partially answered (2026-08-04): it generalizes across labs — GPT-Live ships the same split in production (full-duplex voice model delegating to GPT-5.5), independently derived from latency engineering. Whether the split is permanent or transitional remains open; two implementations are evidence of convergence, not permanence.
    • Wait"Interactivity scales with intelligence" is asserted; the larger-model release later in 2026 is the test.
    • WaitResearch grant announced for interactivity benchmarks — what becomes the FD-bench equivalent for video proactivity?
  • Live-Path Minimalism2 open
    • WaitDoes the upcoming GPT-Live API expose the media/application separation to third parties — application logic customizable behind the async RPC boundary without touching the live path — or is the boundary internal-only? Trigger: GPT-Live API launch.
    • WaitTML upstreamed streaming-sessions serving into SGLang; GPT-Live's stateful serving (persistent sessions, seamless instance handoff, off-path compaction) is proprietary. Does an open-source inference stack ship instance handoff for full-duplex voice? Trigger: an SGLang/vLLM release with session-handoff support.
  • Why AI Lags at Design3 open
    • WaitAre reasons 3–4 (novelty, the abstraction layer) genuine ceilings, or — like reasons 1–2 — just under-invested capabilities that fall once a lab builds the grader?
    • WaitCan design be made gradable without a human in the loop (learned taste models, preference data at scale), or does the "human aspect of taste" resist automation the way research taste might?
    • WaitDoes the design↔code abstraction layer improve with better code-understanding models even if pure visual design stalls — i.e. is reason 4 a coding-capability problem in disguise?