待翻译：Qwen3.8-Flash-Next: How to Run Locally

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

For the complete documentation index, see llms.txt. This page is also available as Markdown. Qwen3.8-Flash-Next is a new open-weight, 125B parameter MoE multimodal model from Qwen. Built on the new Qwen4 architecture, it supports a 262K context window and advanced reasoning. The model outperforms Claude-4.6-Opus (Max) and can run locally on devices with 75GB RAM/unified memory with no GPU VRAM required. To run the model, use our GGUFs with our specific llama.cpp PR for Unsloth Desktop as well. Thank you Qwen for day zero access. 1-bit is 75GB and uses 4-bit for the Ngram / PLE. This is 79% smaller than BF16 (355GB), and retains a top-1% accuracy of 80%. Run Qwen3.8-Flash-Next GuideDownload Unsloth Qwen3.8-Flash-Next is now available to run locally in Unsloth Desktop! ⚙️ Usage Guide Whether you run Qwen3.8-Flash-Next on a CPU with system RAM or on a GPU with VRAM may make relatively little difference. Its unique architecture allows inference using RAM or unified memory to achieve performance closer to that of GPU VRAM than is typical for other models. This makes it particularly well suited to Macs, NVIDIA DGX Spark systems, and other devices with large memory capacities. You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of new Ngram layers or per layer embeddings which is like a lookup table. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models. You can also offload the PLE / Ngram layer to SSD and use mmap which allows less usage of CPU and GPU VRAM. Qwen3.8-Flash-Next Requirements: The smallest quant works on 75GB RAM so it's best to have a 96GB RAM/unified memory device. Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory) 1-bit 2-bit 3-bit 4-bit 5-bit 8-bit BF16 75 GB 79 GB 90 GB 112 GB 200 GB 270 GB 355 GB Recommended Settings Qwen3.8-Flash-Next is a hybrid thinking model with different default settings for thinking and non-thinking modes. Extra high is enabled by default so if you want shorter thinking traces, you can adjust the thinking effort: Parameter Thinking Mode Instruct (non-thinking) Mode temperature 1.0 0.7 top_p 0.95 0.80 top_k 20 20 min_p 0.0 0.0 presence_penalty 0.0 1.5 repetition_penalty 1.0 1.0 Context length = up to 262,144 Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0 💡 Thinking + Preserve Thinking Qwen3.8-Flash-Next has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth has 'Think' and Preserved Thinking toggles for Qwen3.8 (see right): Qwen3.8-Flash-Next comes with support for reasoning_effort, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth: xhigh (default): for complex tasks demanding thorough analysis medium: balancing accuracy and speed low: efficient reasoning optimizing for speed and cost none To change thinking / reasoning effort in unsloth run or llama-server, use --chat-template-kwargs '{"reasoning_effort":"medium"}' If you're on Windows Powershell, use: --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" Change medium to your desired reasoning level. Quantization Analysis We ran KLD for Qwen3.8-Flash quants, and show that 80% top-1% accuracy recovery is possible with 79% less disk space usage. The new architecture uses PLE / Ngrams, and these are not quantized that heavily (4-bit minimum) since they have random access pattern, and quantizing them heavily will damage the model. quant size_gb mean_kld same_top_pct UD-Q4_K_XL 111.3 0.044715 93.481 UD-IQ4_XS 93.7 0.079162 91.089 UD-Q3_K_XL 90 0.099694 90.387 UD-IQ3_XXS 82 0.156505 87.57 UD-Q2_K_XL 78.9 0.213343 85.163 UD-IQ1_M 74.5 0.302159 82.396 UD-IQ1_S 72.5 0.37514 80.239 Run Qwen3.8-Flash-Next Guide You can soon run Qwen3.8-Flash-Next in Unsloth Desktop and llama.cpp with our specific PR. Feel free to change quantization type. Hugging Face: Qwen3.8-Flash-Next-GGUF ModelScope: Qwen3.8-Flash-Next-GGUF Run in Unsloth DesktopRun in llama.cpp Qwen3.8-Flash-Next is now available to run locally in Unsloth Desktop! 🦥 Run Qwen3.8-Flash-Next in Unsloth Qwen3.8-Flash-Next now is able to run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux and: Search, download, run GGUFs and safetensor models Self-healing tool calling + web search Code execution (Python, Bash) Automatic inference parameter tuning (temp, top-p, etc.) Fast CPU + GPU inference via MLX and llama.cpp Train LLMs 2x faster with 70% less VRAM 1 Install Unsloth The easiest way to get started is by downloading the Unsloth Desktop app. Works on macOS, Windows, and Linux. Download Unsloth Download for macOS Download for Windows Download for Linux Or, if you prefer to install manually: MacOS, Linux, WSL: Windows PowerShell: 2 Search and download Qwen3.8-Flash-Next Go to Unsloth Chat or Model hub and search for Qwen3.8-Flash in the search bar and download your desired model and quant. 3 Run Qwen3.8-Flash-Next Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings. For more information, you can view our Unsloth inference guide. For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more. Dynamic 1-bit 397GB 91% smaller GGUF of Qwen3.8 2.4T in Unsloth Desktop 4 Serve Qwen3.8-Flash-Next with Unsloth API You can use unsloth run command and serve Qwen3.8 via an API using llama-server runtime flags, including context sizing, GPU layers, threading, sampling, networking, and tool configuration. For more info see our API docs or unsloth start. 5 Unsloth is now ready You can also do many other things with Qwen3.8-Flash-Next via Unsloth Desktop like: Connect tools: Claude Code, Codex, web search, MCP and more Train models: Fine-tune text, diffusion, embedding, and more Generate media: Create and train images, video, TTS locally 🦙 Run Qwen3.8-Flash-Next in llama.cpp 1 We need to use our specific llama.cpp PR here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default. 2 To run the model, you can do: 3 Then to run it: 📊 Benchmarks For GGUF quantization benchmarks you can see above for our quantization analysis or Dynamic V3.0 article. PreviousModel Export NextGLM-5.3-Flash Last updated 13 minutes ago Was this helpful?