Beehiiv adds Cloudflare AI Crawl Control so writers can block or allow bots
Cloudflare and beehiiv integrate AI Crawl Control into the beehiiv dashboard, giving independent publishers a simple toggle to allow or block AI crawlers. The feature addresses the challenge of AI bots consuming increasing web traffic and provides transparency into referral traffic trade-offs.
Cloudflare Inc. and newsletter platform beehiiv Inc. today launched an integration that hands independent publishers a single toggle to decide whether artificial intelligence crawlers can reach their work.
The integration bakes Cloudflare’s AI Crawl Control into the beehiiv dashboard. Writers face a choice. They can open their archives to AI search engines and agents and chase the distribution that brings, or shut AI scraping off and hold their content back for licensing deals later. Beehiiv is turning the option on for its users starting today.
Behind the launch is a problem that has gotten worse fast. AI systems now eat up a large and rising share of web traffic and fending them off has meant fiddling with robots.txt files or writing firewall rules. That is work most solo creators were never going to do. Putting the controls in the dashboard takes the code out of it.
Flip the controls on and a publisher gets a dashboard, built on Cloudflare’s application programming interfaces, that names the AI crawlers knocking on the door. It shows what is getting blocked.
It also shows how much referral traffic those crawlers are sending back, the trade-off at the heart of the open-or-block decision. Want to block one model and wave another through? That is a click. Cloudflare says the list keeps itself current as fresh crawlers turn up.
The deal is the latest in a year-long campaign. Cloudflare flipped on default AI-scraper blocking for new customers in 2025 and stood up a Pay Per Crawl marketplace where publishers can charge AI firms for access. In January, it bought AI data marketplace startup Human Native Ltd. to flesh out licensed-content tooling for rights holders.
Cloudflare has framed the campaign as a fight for content creators “from independent bloggers to the world’s largest publishers,” in the words of co-founder and Chief Executive Matthew Prince. He called the beehiiv partnership the next step in giving newsletter operators the transparency and control to set their own terms with AI companies, whether they want discovery or want to hold their work back.
Tyler Denk, beehiiv’s co-founder and CEO, sees the integration as a bargaining chip for writers. AI is changing how readers stumble onto content, he said and creators ought to have a hand on that lever.
Some 135,000 publishers run newsletters, websites and podcasts on beehiiv. It takes nothing from their subscription revenue and the audiences stay theirs. For Cloudflare, the integration buys reach into the creator economy and another on-ramp for the Pay Per Crawl model it is wagering will change how AI companies pay for what they take.
Image: SiliconANGLE/Ideogram
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media