Inference Cost

Recent conditional video generation models can convert 3D engine renderings like depth maps into photorealistic videos, but suffer from appearance inconsistencies when the camera revisits a location after context eviction in long-horizon autoregressive generation. This paper proposes a training-free method that leverages correspondences from the 3D engine: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence biases token-level attention toward geometrically corresponding regions. Evaluations on loop-closure trajectories from TartanAir and TartanGround show improved revisit consistency without sacrificing video quality, outperforming existing training-free baselines.

Black Forest Labs (BFL) releases FLUX 3, a multimodal foundation model that learns from images, videos, and audio within a single architecture. It is the first FLUX model to output video, audio, and action predictions from one set of weights. The model builds on the Self-Flow method and excels in video generation, producing clips up to 20 seconds with native audio. In human preference tests, FLUX 3 outperforms many competitors. The same backbone also drives the FLUX-mimic robot policy with sub-80 ms latency.

Datalab released Marker 2, a full rewrite of its open-source document conversion pipeline. It scores 76.0% on olmOCR-bench in balanced mode, sustaining 2.9 pages per second on a single B200 GPU—over 5× MinerU's pipeline throughput—while outperforming Docling on both accuracy and speed. The article compares Marker 2 with MinerU, Docling, and LiteParse across performance, licensing, and use cases.

Anthropic's latest Claude upgrade targets developers and enterprises with stronger coding, better reasoning efficiency, prompt-cache-friendly tool changes, and near-Fable performance at Opus pricing.

As language model parameter sizes grow, effective compression is essential to reduce computational and memory overhead. Existing methods suffer from performance degradation at high compression ratios. This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal—their combination causes significant performance loss. The authors propose Diagonal Adhesive Method (DAM) to effectively combine both techniques and mitigate the loss.

Large language models (LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time rather than during prefill. We propose DecodeShare, a protocol that identifies a low-dimensional subspace consistently shared across tasks in decode-time hidden states, and then tests its causal role by removing that subspace only during decoding. In our experiments, disturbing the discovered shared subspace degrades decision performance far more than disturbing either a prefill-derived or random subspace under the same intervention budget. We further show this decode-shared subspace has practical consequences for activation steering: common steering directions can overlap the task-general decode channel. Projecting out this shared subspace directly separates the functional roles of the two components, while evaluating steering vectors at decode-time yields more reliable signal for downstream deployment than prefill-based proxies. Despite its compactness, the shared subspace can serve as a high-leverage causal channel at decode time.

InferenceBench evaluates AI agents on open-ended LLM inference optimization. Agents must deploy an inference server and optimize speed under a two-hour budget across four scenarios: prefill latency, decode latency, concurrent throughput, and a balanced mix. Agents improve up to 8x over baselines but fall short of simple hyperparameter search (11.53x). Analysis reveals agents converge on a single framework, lacking diverse exploration.

DC-Leap is a training-free framework that accelerates diffusion large language models (dLLMs) by addressing the Joint Probability Dependence Error (JPDE) that leads to overly conservative confidence thresholds. It introduces Dynamic Contiguous Verification and draft-guided decoding to achieve up to 53.19x speedup on MBPP long-sequence generation and up to 105.02x when combined with KV-Cache, while maintaining generation quality.

NVIDIA's senior director of generative AI software, Joey Conway, discusses how local open models are increasingly working alongside frontier models, with routers deciding which to use, enabling organizations to achieve better outcomes at lower cost and latency.

VQ-Transplant introduces a lightweight framework for plug-and-play integration of new vector quantization modules into frozen pre-trained tokenizers without costly end-to-end retraining. A lightweight decoder adaptation trained for only 5 epochs on ImageNet-1k mitigates quantization mismatch, achieving near state-of-the-art reconstruction fidelity on industry-level models like VAR while reducing training cost by 95%. This democratizes quantization research, enabling resource-efficient exploration of novel VQ techniques.

NEXUS is a structured-plan safety monitor that combines deterministic safety rules, argument-level inspection, and a calibrated logistic-regression risk score to allow, block, request confirmation, or request revision for LLM agent actions. It achieves strong benchmark results with minimal latency.

A new study benchmarks the performance cost of enabling confidential computing for LLM inference on an NVIDIA H100 GPU under Intel TDX. Using Mistral-7B and Qwen3-30B-A3B models, results show a 21.8%-27.8% increase in time-to-first-token and 17.7%-21.1% drop in global token throughput in confidential mode. The larger model reaches saturation earlier, highlighting the need for capacity planning adjustments.

A hierarchical LLM framework combining cloud-based and edge LLMs with DRL for UAV navigation in ITNTNs, reducing collisions and improving throughput.

The paper identifies a robustness gap introduced by post-training quantization (PTQ) in robotic perception models deployed on edge devices. While PTQ maintains in-distribution accuracy, it reduces reliability under distribution shifts. The authors propose Recti-Q, a lightweight feature-space rectification method that uses a frozen quantized backbone and a small LoRA adapter, achieving significant robustness recovery with minimal overhead.

Surprise Forcing is a training-free framework that improves long video generation by addressing two limitations of streaming autoregressive diffusion: bounded context and fixed denoising schedule. It uses a Surprise-Gated Memory Bank to selectively retain important visual evidence and Surprise-Aware Denoising to skip denoising steps for easy chunks. Experiments show improved consistency and quality while maintaining real-time throughput.

Modern LLM query routers often ignore generation latency, focusing only on accuracy and cost. This paper introduces a lightweight latency estimator that simulates autoregressive token batch processing to predict time-to-first-token (TTFT), and integrates it into a router that jointly optimizes latency, accuracy, and cost. Experiments show up to 40% improvement in accuracy-cost utility while maintaining the same latency as standard load-balancing approaches.

As AI models integrate into critical systems, existing software safeguards may be bypassed. Researchers propose microarchitecture knobs that dynamically control GPU memory subsystem resources (L2 cache size, latency, bandwidth, shared memory port access rate) to limit AI performance at runtime, achieving up to 80% performance reduction with negligible cost.

This article examines the poor tokenomics of local AI development, where running a single agent on expensive hardware yields low throughput. It introduces agent swarms—parallel task execution across many agents—as a game-changer. By saturating GPUs with parallel workloads, local hardware becomes cost-effective compared to API calls. Detailed calculations show that a 32-agent swarm on a local rig costs only a fraction of API-based alternatives, making local AI worthwhile for the first time.

This tutorial explores NVIDIA's srt-slurm framework, learning how to use srtctl to convert declarative YAML configurations into reproducible SLURM benchmark workflows for distributed LLM serving. We set up the project in Google Colab, inspect its internal architecture, define a cluster configuration, dry-run built-in and custom recipes, and model a disaggregated prefill-and-decode deployment for DeepSeek-R1. We also generate parameter sweeps, interact with the typed Python API, validate expanded configurations, and analyze simulated benchmark results through a throughput-versus-latency Pareto frontier.

Google has released Gemini 3.6 Flash and 3.5 Flash-Lite as new workhorses designed to cut latency and token costs for enterprise AI agents. The new models offer significant performance improvements, targeted pricing, and integrated computer-use tools, with enterprise partners already deploying them in production.

NVIDIA Vera Rubin NVL72 production is ramping up with partners CoreWeave, Google Cloud, Microsoft Azure and Oracle Cloud Infrastructure. The platform delivers highest performance per watt and lowest token cost, with 10x more throughput per megawatt than Grace Blackwell NVL72 in benchmarks. It also powers Europe's open-model era through a partnership between Microsoft and Mistral.

Google released Gemini 3.6 Flash, a cheaper and faster 3.5 Flash-Lite, and 3.5 Flash Cyber, but the flagship 3.5 Pro remains delayed. 3.6 Flash shows significant improvements in benchmarks and lower output costs. 3.5 Flash-Lite targets high-throughput tasks with strong cost-performance. 3.5 Flash Cyber, for cybersecurity, matches Opus 4.6 but is limited to pilot access.

Prince Canuma, creator of MLX-VLM, launches Nativ, a macOS desktop app that wraps MLX with a chat interface and local API server, automatically detecting models in your Hugging Face cache.

This paper presents ARGO, a smart eyewear platform leveraging the STM32N6 microcontroller and its integrated NPU for on-device machine learning, minimizing latency and preserving privacy. Through holistic co-design of hardware, firmware, and AI, an optimized YOLOv11 model is deployed for real-time urban obstacle recognition, introducing Head-wise Parallel Attention (HPA) for efficient NPU execution. The model achieves mAP50-95 of 24 with only 2.483 MB memory footprint. The platform integrates multimodal sensors, runs at 10 FPS, and provides ~113 minutes of autonomy on a 200 mAh battery, demonstrating the feasibility of high-performance, privacy-preserving assistive devices.

arXiv:2607.16200 presents agrepl, a CLI framework for deterministic replay of AI agent executions. Using a MITM proxy, it records external interactions and replays them in isolation, achieving perfect fidelity (F=1.0) and 98.3% latency reduction.

Moonshot AI’s Kimi K3, a 2.8-trillion-parameter open-weight model, uses mixture-of-experts, quantization, and attention caching to trade compute for memory, circumventing US chip restrictions. While it tops benchmarks in coding, deployment requires data-center infrastructure, pricing is high, and software support is incomplete.

This week's highlights include stopping if-else chains with the registry pattern, 12 ways to reduce LLM latency and costs, 5 real-world SQL projects for your portfolio, Git worktrees for AI development, structured generation with Outlines, 7 Python frameworks for local AI agents, 10 YouTube channels to stay ahead in AI, getting started with Conductor for Gemini CLI, 5 free resources on agentic AI, and working with Pi coding agents.

Meta's new Muse Spark 1.1 model is now available on Databricks via Model Provider Services (MPS) in Unity AI Gateway. This service allows organizations to register providers once in Unity Catalog, eliminating API key sprawl and centralizing governance through familiar permissions, rate limits, and guardrails. Additionally, every request is automatically tracked with token usage, latency, cost attribution, and audit logs for end-to-end observability.

NVIDIA released Nemotron 3 Embed on July 15 and 16, 2026. The collection has three open checkpoints: Nemotron-3-Embed-8B-BF16, Nemotron-3-Embed-1B-BF16, and Nemotron-3-Embed-1B-NVFP4. The 8B ranks #1 on RTEB at 78.46 average NDCG@10. The 1B came from ModelOpt NAS pruning plus COS+MSE distillation from the 8B teacher. NVFP4 retains 99%+ of BF16 retrieval accuracy at up to 2x Blackwell throughput. All three run 32,768-token inputs under OpenMDW-1.1.

Polestar is a training-free inference framework that addresses KV-cache reuse and decoding parallelism challenges in diffusion LLMs by leveraging token representation drift. It consists of Polestar-Cache for sparse cache refreshes and Polestar-Commit for identifying commit-ready tokens, achieving up to 10.73% accuracy improvement and 3.7x higher throughput on math and coding benchmarks.

A multimodal teleoperation architecture for ROVs using 3D Gaussian Splatting to generate occlusion-free exocentric views and a vibrotactile suit for haptic cues. A human study with 30 participants showed the exocentric view significantly improves performance under high latency, with fNIRS indicating sustained executive control rather than cognitive overload.

This work fine-tunes LLaMA 3 (8B) as an efficient drop-in reranker via supervised fine-tuning and 4-bit quantization, replacing cross-encoders in RAG pipelines. It achieves 14-21% improvement in answer relevancy, context precision, answer similarity, and answer correctness on a domain-specific QA benchmark while reducing inference overhead.

Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory. This paper introduces Semidirect Fourier Delta Attention (SFDA), a phase-controlled generalization of Kimi Delta Attention that replaces real diagonal decay with block-rotational Fourier control. The main result is a constructive chunk-WY factorization, enabling exact affine chunk transfer, formal stability and complexity bounds, and a compact characterization of phase-plus-low-rank memory. Experiments show SFDA learns cyclic memory while the phase-disabled KDA baseline remains near chance.

The State of Open Source AI report reveals that open-weight models have achieved near-parity with closed models in capability, while inference costs dropped 50x in 36 months. Open models are adopted by 79% of developers but only 51% reach production due to operational challenges. The report emphasizes open source as a sovereignty choice, with over 70 national AI strategies in place.

Scaling LLMs isn’t about adding GPUs. It’s about removing wasted work from every request.

This paper proposes a human-efficient post-training pipeline that enables a small number of human operators to supervise multiple robots through specialized division of labor and automatic trajectory segmentation using VLAC-CUT. Validated on four real-world manipulation tasks, the final policies achieve 80%-95% success rates and improve task throughput by 1.7x-4.2x over the base model.

A new study shows that post-training quantization can silently alter how large language models reason even when task accuracy is preserved. Using a six-category failure taxonomy, the researchers classified 30,000 chain-of-thought outputs and found that hollow convergence exhibits a size-dependent shift under NF4 quantization, while shortcut collapse and confidence snowballing undergo qualitative changes. Hollow convergence cannot be reliably detected from surface-level text features, posing a deployment risk.

This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. The authors replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, perform embedding calibration and fine-tuning, achieving a 59.2% tie-excluded win rate against Google Translate on a subset of OpenSubtitles2024, and a 1.63x speedup on Apple M2.

A new paper introduces MawForge, a system that enables practical local inference of Sparse Mixture-of-Experts (MoE) language models on memory-constrained unified-memory machines by storing the model on disk and materializing expert tensors on demand into a bounded cache. The system is effective as a measurement substrate but not as a cache-maximization policy.

This paper presents a closed-loop control framework using a small language model (SLM) aligned via Group Relative Policy Optimization (GRPO). The system integrates an action agent, a digital-twin validator, and a reprompting agent to iteratively correct outputs. In thermal control simulations, it achieves 91.5% action-alignment accuracy with 3.84s inference latency, demonstrating viability for edge autonomous control.

OpenAI's Codex reaches 7M users, adding 1M in a day, with 10x growth in 6 months. Prime Intellect releases verifiers v1 for agent RL. OpenAI transparently fixes GPT-5.6 Sol usage issues. Grok Build security controversy emerges. Open models and quantization progress. Continual learning research resurfaces.

PlanWright is a control plane for AI coding agents that inverts planning and acceptance ceremonies to eliminate human bottlenecks, delivering agent-speed throughput with cryptographic audit trails.

Director is a new distributed MoE serving system that minimizes end-to-end latency through prediction-driven, online expert placement. It uses a lightweight cascaded predictor or low-bit quantized replica for expert activation patterns, an online migration module with near-zero downtime, and a relaxation-based optimizer that achieves a (1+ε) approximation ratio in polynomial time. Experiments show an 11–55% reduction in latency for popular MoE models.

This paper introduces signed symmetric quantization for few-bit integers, addressing clipping errors from standard symmetric quantizers while avoiding the runtime penalty of asymmetric quantization. The method places the extra negative value on the dominant outlier tail, achieving better perplexity and accuracy on large language models at no extra inference cost.

KV-PRM is an efficient process reward model that eliminates text re-encoding by directly using the KV cache from LLM generation, reducing scoring cost from O(L²) to O(L). It matches or outperforms text-PRMs on benchmarks with up to 5000x FLOPs reduction, 37x latency reduction, and 34x memory reduction.

AI performance depends on three dimensions: accuracy, throughput, and interactivity. This post focuses on throughput and interactivity, examining how model-design choices can optimize both without sacrificing accuracy, aiming to push the Pareto frontier outward.

An exploration of the inference process in large language models, covering autoregressive generation, prefill and decode phases, the KV cache, and decoding strategies, explaining the mechanics behind token-by-token output.

Tensorlake rebuilt sandbox ingress, moving from L7 reverse proxy to L4 byte forwarding using kernel TLS (kTLS) and splice(2) for zero-copy data paths, achieving 2.2x throughput and halving CPU cost. The new architecture decouples the data plane from the control plane, uses kTLS for in-kernel crypto, and derives adaptive timeouts from byte flow. Performance tests show single-connection throughput increases from 1.12 GB/s to 2.50 GB/s, with proxy CPU per GB dropping from 0.90 to 0.49 CPU-seconds.

Henry Schein One developed Image Verify, an AI-powered system on Amazon SageMaker AI that evaluates dental X-ray quality in real time, reducing insurance claim denials. The system scaled from concept to over 10,000 locations in months, processing millions of X-rays with sub-2-second latency.

Learn four deployment patterns for deploying Unsloth-quantized models on AWS: using EC2 for direct access, SageMaker AI for managed serving, and EKS/ECS for containerized inference. Understand Unsloth's dynamic quantization, model formats (GGUF, safetensors), and operational best practices.