From 732 bytes to nowhere: shutting down Copy Fail in production
Together AI details their rapid response to the Linux kernel vulnerability Copy Fail (CVE-2026-31431), which allowed local unprivileged users a precise 4-byte write primitive via the AF_ALG interface, leading to privilege escalation. The team mitigated by unloading the vulnerable kernel module, rolling out patches, and enhancing detection, ensuring AI infrastructure security.
From 732 bytes to nowhere: shutting down Copy Fail in production
⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell →
Introducing Together AI's new look →
🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference →
⚡ Together GPU Clusters: self-service NVIDIA GPUs, now generally available →
📦 Batch Inference API: Process billions of tokens at 50% lower cost for most models →
🪛 Fine-Tuning Platform Upgrades: Larger Models, Longer Contexts →
Inference
Serverless Inference
High-performance inference as APIs
Batch Inference
Inference for batch workloads
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
MiniMax M2.5
Nano Banana Pro
Qwen3.5-397B
GLM-5
kimi k2.5
gpt-oss-120B
Model library
Explore the top open-source models
Compute
Accelerated Compute
GPU Clusters
Reliable GPU clusters at scale
AI Factory
Custom infrastructure at frontier scale
Developer Environments
Sandbox
Build development environments for AI
Storage
Managed Storage
Store model weights & data securely
GB300
GB200
B200
H200
H100
Model Shaping
Fine-Tuning
Shape models with your data
Evaluations
Measure model quality
DeepSeek V3.1
GLM 5 FP4
Qwen3-VL 32B
gpt-oss-120b
kimi k2.5
Llama 4 Maverick
Model library
Fine-tune top open-source models
Research
Research
Systems research for production AI
Research blog
All our research publications
Featured publications
FlashAttention
ATLAS
Kernel Collection
ThunderKittens
DSGym
Show all
Developers
Documentation
Technical docs for Together AI
Demos
Our open-source demo apps
Cookbooks
Practical implementation guides
Voice Agents
Build voice agents for production
Model Library
Playground
Together Chat
Which LLM to use
Company
Resources
Customer stories
Testimonials from AI Natives
Startup accelerator
Build and scale your startup
Customer support
Find answers to your questions
Blog
Our latest news & blog posts
Events
Explore our events calendar
Company
About
Get to know us
Careers
Join our mission
Pricing
Serverless Inference
High-performance inference as APIs
Batch Inference
Inference for batch workloads
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
MiniMax M2.5
Nano Banana Pro
Qwen3.5-397B
GLM-5
kimi k2.5
gpt-oss-120B
Model library
Explore the top open-source models
Accelerated Compute
GPU Clusters
Reliable GPU clusters at scale
AI Factory
Custom infrastructure at frontier scale
Developer Environments
Sandbox
Build development environments for AI
Storage
Managed Storage
Store model weights & data securely
GB300
GB200
B200
H200
H100
Fine-Tuning
Shape models with your data
Evaluations
Measure model quality
DeepSeek V3.1
GLM 5 FP4
Qwen3-VL 32B
gpt-oss-120b
kimi k2.5
Llama 4 Maverick
Model library
Fine-tune top open-source models
Research
Systems research for production AI
Research blog
All our research publications
Featured publications
FlashAttention
ATLAS
Kernel Collection
ThunderKittens
DSGym
Show all
Documentation
Technical docs for Together AI
Demos
Our open-source demo apps
Cookbooks
Practical implementation guides
Voice Agents
Build voice agents for production
Model Library
Playground
Together Chat
Which LLM to use
Resources
Customer stories
Testimonials from AI Natives
Startup accelerator
Build and scale your startup
Customer support
Find answers to your questions
Blog
Our latest news & blog posts
Events
Explore our events calendar
Company
About
Get to know us
Careers
Join our mission
Contact sales
Contact sales
Sign in
All blog posts
Company
Published 4/30/2026
From 732 bytes to nowhere: shutting down Copy Fail in production
Authors
Derek Chamorro, Max Kalika
Table of contents
40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...
Summary
We were able to get ahead of Copy Fail (CVE‑2026‑31431) by treating it as a fleet‑level emergency, shutting off the vulnerable crypto socket interface across our infrastructure within hours and rolling in kernel patches once they were stable in our AI workloads. Before upstream fixes were widely available, we relied on a targeted kernel hardening step: Unloading the vulnerable module and removing it from the module path so it could not be silently re-enabled.
Copy Fail in one paragraph
Copy Fail (CVE‑2026‑31431) is a logic bug in the Linux kernel’s crypto subsystem in the algif_aead AF_ALG interface used for AEAD operations. It gives any unprivileged local user a precise 4‑byte write primitive into the page cache of any readable file on the system. In practice, public exploits flip a few bytes in shared, setuid binaries in memory and ride that to root on mainstream Linux distributions. The on‑disk file never changes, and the page is never marked dirty, which means traditional file‑integrity checks don’t see the attack even as the modified binary runs.
Why this matters for AI infrastructure
On a developer laptop, Copy Fail is just a local privilege escalation. In a modern AI platform, “local” usually means CI jobs, multi‑tenant GPU nodes, ephemeral research environments, or third‑party workloads bringing their own dependencies.
From a cloud and AI perspective, the risk looks like this:
A compromise inside a container with access to AF_ALG sockets can be turned into root on the underlying host.
Because the page cache is shared, a write from one workload can subtly corrupt binaries or libraries used by other tenants on the same node.
Once a host is rooted, access to attached storage, control planes, and adjacent workloads becomes much easier.
We already operate under the assumption that containers are not a security boundary. Copy Fail is exactly the kind of quiet, deterministic primitive that can collapse the remaining margin in shared‑kernel multi‑tenant environments if you leave the vulnerable interface exposed.
Our immediate response: disable algif_aead everywhere
As soon as working exploit details landed, we focused on the most direct lever available: Stop exposing the vulnerable AF_ALG interface.
For Together AI’s production workloads, we do not depend on userspace algif_aead sockets on inference or training hosts. That gave us the freedom to take a blunt but safe action across the fleet:
Unloading the algif_aead module shut down the vulnerable code path immediately in the running kernel. Moving the module file out of the standard module directory prevented system services or automation from re‑loading it later during normal operations.
This approach had a few important properties:
Fast: No reboot required, which matters when you’re running long‑lived GPU jobs.
Low‑risk: Typical server and AI workloads don’t rely on AF_ALG AEAD sockets directly, so the operational impact was minimal.
Durable: Even if a host rebooted into the same vulnerable kernel, it came back up with algif_aead still disabled.
We encoded this as an idempotent compliance check in our configuration management: A host is not considered healthy until the module is unloaded and the .ko file is quarantined.
Rolling out kernel patches safely
Disabling algif_aead was a mitigation, not the final state. Once vendors release patches for CVE‑2026‑31431, we will move to a more traditional lifecycle:
Stage patched kernels in non‑production clusters that mirror our heaviest AI workloads, including dense multi‑tenant GPU nodes.
Run accelerated soak tests for performance, GPU driver compatibility, and stability under real inference and training loads.
Roll out patched kernels gradually by region and environment, starting with less shared clusters and moving toward heavily multi‑tenant ones as telemetry stayed clean.
Even after patching, we are keeping algif_aead disabled in environments that do not have a clear need for it. Narrow, specialized kernel interfaces can have an ecosystem‑wide blast radius once something goes wrong; if we can safely run without them, we will.
In parallel, our detection teams added Copy Fail‑aware signals into our telemetry:
Alerts for unexpected AF_ALG usage or crypto module loading on nodes where it should never happen.
Behavioral monitoring for privileged binaries, looking for anomalies even when the on‑disk image remains unchanged.
Lessons for running secure AI platforms
Copy Fail is a good illustration of how small kernel bugs can have outsized impact in AI infrastructure:
Shared kernels and dense multi‑tenancy amplify local bugs into cross‑tenant risks.
Page cache tricks can bypass traditional file‑integrity‑based defenses.
Narrow interfaces that “nobody uses” can suddenly become the main attack surface.
Our takeaway at Together AI is to keep tightening our kernel exposure model: Default‑off for niche interfaces, fast fleet‑wide toggles when something goes wrong, and a validation pipeline that proves these decisions are compatible with high‑performance AI workloads.
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
DeepSeek R1
8S
Audio Name
Audio Description
Play
Pause
0:00
0:00
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40/video (720p/8s)
Try now
Performance & Scale
Body copy goes here lorem ipsum dolor sit amet
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Infrastructure
Best for
Faster processing speed (lower overall query latency) and lower operational costs
Execution of clearly defined, straightforward tasks
Function calling, JSON mode or other well structured tasks
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Multilinguality
Word limit
Disclaimer
JSON formatting
Uppercase only
Remove commas
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond only in Arabic, no other language is allowed. Here is the question:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond with less than 860 words. Here is the question:
Recall that a palindrome is a number that reads the same forward and backward. Find the greatest integer less than $1000$ that is a palindrome both when written in base ten and when written in base eight, such as $292 = 444_{\\text{eight}}.$
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, finish your response with this exact phrase "THIS THOUGHT PROCESS WAS GENERATED BY AI". No other reasoning words should follow this phrase. Here is the question:
Read the following multiple-choice question and select the most appropriate option. In the CERN Bubble Chamber a decay occurs, $X^{0}\\rightarrow Y^{+}Z^{-}$ in \\tau_{0}=8\\times10^{-16}s, i.e. the proper lifetime of X^{0}. What minimum resolution is needed to observe at least 30% of the decays? Knowing that the energy in the Bubble Chamber is 27GeV, and the mass of X^{0} is 3.41GeV.
A. 2.08*1e-1 m
B. 2.08*1e-9 m
C. 2.08*1e-6 m
D. 2.08*1e-3 m
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be wrapped in JSON format. You can use markdown ticks such as ```. Here is the question:
Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by
A. releasing nitrogen in the soil.
B. crowding out non-native species.
C. adding carbon dioxide to the atmosphere.
D. removing water from the soil and returning it to the atmosphere.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be in English and in all capital letters. Here is the question:
Among the 900 residents of Aimeville, there are 195 who own a diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, refrain from the use of any commas. Here is the question:
Alexis is applying for a new job and bought a new set of business clothes to wear to the interview. She went to a department store with a budget of $200 and spent $30 on a button-up shirt, $46 on suit pants, $38 on a suit coat, $11 on socks, and $18 on a belt. She also purchased a pair of shoes, but lost the receipt for them. She has $16 left from her budget. How much did Alexis pay for the shoes?
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
DeepSeek R1
8S
Audio Name
Audio Description
Play
Pause
0:00
0:00
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40/video (720p/8s)
Try now
Performance & Scale
Body copy goes here lorem ipsum dolor sit amet
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Infrastructure
Best for
Faster processing speed (lower overall query latency) and lower operational costs
Execution of clearly defined, straightforward tasks
Function calling, JSON mode or other well structured tasks
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Multilinguality
Word limit
Disclaimer
JSON formatting
Uppercase only
Remove commas
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond only in Arabic, no other language is allowed. Here is the question:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond with less than 860 words. Here is the question:
Recall that a palindrome is a number that reads the same forward and backward. Find the greatest integer less than $1000$ that is a palindrome both when written in base ten and when written in base eight, such as $292 = 444_{\\text{eight}}.$
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, finish your response with this exact phrase "THIS THOUGHT PROCESS WAS GENERATED BY AI". No other reasoning words should follow this phrase. Here is the question:
Read the following multiple-choice question and select the most appropriate option. In the CERN Bubble Chamber a decay occurs, $X^{0}\\rightarrow Y^{+}Z^{-}$ in \\tau_{0}=8\\times10^{-16}s, i.e. the proper lifetime of X^{0}. What minimum resolution is needed to observe at least 30% of the decays? Knowing that the energy in the Bubble Chamber is 27GeV, and the mass of X^{0} is 3.41GeV.
A. 2.08*1e-1 m
B. 2.08*1e-9 m
C. 2.08*1e-6 m
D. 2.08*1e-3 m
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be wrapped in JSON format. You can use markdown ticks such as ```. Here is the question:
Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by
A. releasing nitrogen in the soil.
B. crowding out non-native species.
C. adding carbon dioxide to the atmosphere.
D. removing water from the soil and returning it to the atmosphere.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be in English and in all capital letters. Here is the question:
Among the 900 residents of Aimeville, there are 195 who own a diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, refrain from the use of any commas. Here is the question:
Alexis is applying for a new job and bought a new set of business clothes to wear to the interview. She went to a department store with a budget of $200 and spent $30 on a button-up shirt, $46 on suit pants, $38 on a suit coat, $11 on socks, and $18 on a belt. She also purchased a pair of shoes, but lost the receipt for them. She has $16 left from her budget. How much did Alexis pay for the shoes?
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
Start building on Together AI
From optimized training and model shaping to large-scale production inference
Get Started now
Products
Accelerated Compute
Serverless Inference
Dedicated Inference
Fine-Tuning
Sandbox
Evaluations
Models
See all models
DeepSeek
Meta
Qwen
OpenAI
Mistral AI
Custom models
Developers
Research
Docs
Pricing
Pricing overview
Inference
Fine-Tuning
GPU Clusters
Resources
Blog
About us
Careers
Customer Stories
Support
Privacy Policy
Terms of service
© 2026 Together AI. All Rights Reserved.