Bot Preference Sync admits robots.txt never enforced

Cloudflare's 21 August 2026 post is the receipt. Jin-Hee Lee's Bot Preference Sync announcement splits the world into two layers. One states a preference and assumes best intent. The other actually blocks with Bot Management.

The money sentence is not the dashboard screenshot.

"When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules."

A paper CLOSED sign on an unlocked door. The crawler reads the gap, not the ink. Sync exists to take that gap away.

A Disallow the edge ignores is a grant#

The product is small. It is also the quiet part out loud.

You already pick Search, Agent, and Training in the zone dashboard. Search and Agent each get three settings.

• Allow, if you want the crawler
• Block only on pages that serve ads
• Block everywhere when you mean it

Training gets a Disallow that writes a no-training preference into the file so a cooperating mixed-use crawler can still index for search.

Bot Preference Sync prepends that policy onto whatever robots.txt already lives at the origin. New customers get it on by default. Free through Enterprise. Turn it off if the file has to stay handmade.

Cloudflare will not read your clever custom WAF rules into that prepend. Category-wide policy only. Existing customers on the old managed robots.txt get a prompt. The bots changelog still has no row. The post says the coming week.

What the post will not let you miss is the failure mode. A crawler Disallowed in the file, allowed at the edge. Some operators treated that mismatch as a basis to disregard the preference. You did not open the door. The missing edge block did.

If you wanted a blocking tutorial, the four-layer writeup already exists. This is the vendor explaining why a mismatch was being used as an argument to keep crawling.

The file was never a lock#

RFC 9309 said it in 2022. "These rules are not a form of access authorization." The same RFC warns that listing paths in robots.txt makes those paths public. A Disallow is a sign on the lawn. It is also a map.

Google's docs are blunter. Search Central's robots.txt intro says the file is for crawl traffic, mainly so you do not melt the origin. It is not a mechanism for keeping a page out of Google. Instructions cannot enforce crawler behavior. A disallowed URL can still show up if other sites link to it.

Cloudflare's managed robots.txt docs repeat the sentence operators have been ignoring for two years. "robots.txt compliance is voluntary." Want a block? That is AI Crawl Control. The file asks. The edge has to answer.

This site is a cheap check, not a sermon. A GET of rizz.dev/robots.txt on 22 August 2026 returned 379 bytes, server: cloudflare, and no # BEGIN Cloudflare Bot Preference Sync markers. Eleven named AI crawlers sit in one group.

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot. All of them Allow: /. Only /api/ is Disallowed. The origin Worker in this repo hard-codes that list. The zone is on Cloudflare and still publishes permission.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
Disallow: /api/

Public posts stay crawlable on purpose. /api/ is the line. Sync would only prepend a Training Disallow if the dashboard said so, and this zone is not saying so.

Anyone who treated a Disallow as a lock was already wrong. Sync does not change that physics. It stops the statement and the latch from telling two different stories.

They already sold the lock as a second click#

Take the steel-man. Cloudflare did not just now discover that the file is an honor system.

In July 2024 they shipped one-click Block AI Bots. In December 2024, Robotcop added an "Enforce robots.txt rules" button. Press it and the AI-bot Disallows in the file become a WAF rule. Celso Martinho's post is explicit. Once you deploy that rule, you are no longer requesting respect. You are enforcing it.

July 2025, Cloudflare's managed robots.txt post called the file an honor system in those words. "Nothing forces bots to follow it." Only 37% of the top 10,000 domains on Cloudflare even had a robots.txt. GPTBot was Disallowed in 7.8% of those files. Google-Extended in 5.6%.

A million-plus customers had flipped the one-click block. The preference layer was still a mess.

So why Sync.

Because the old product ran file into WAF, and only if you remembered the second click. The new product runs dashboard into file, so the public statement matches what the edge already does. Opposite arrow. Same door. You can still have a handmade robots.txt that Sync never sees, because a custom rule is not a category toggle. Cloudflare said that part out loud too.

You'll still need the WAF. A prepend that says Disallow does not drop packets. A crawler that never fetches robots.txt does not care what Cloudflare wrote at the top. Sync kills the excuse that the sign and the latch disagreed. It does not become the latch.

That is the cost of holding this take. If Cloudflare later shows the disagreement was rare, the spicy sentence gets smaller. Until then the 21 August post is the vendor naming the mismatch as the reason to ship Sync.

Mixed-use crawlers buy the verified badge with paperwork#

The part that will actually bite you is not the prepend. It is Training.

Bots that do both Search and Training stay unblocked when you Disallow Training only if they pay in transparency.

• Honor a no-training preference, somehow
• An opt-out of AI summaries
• URL-level visibility into what was trained on, plus search metrics
• Public proof that Disallowing Training does not tank classic search

Skip the paperwork and they lose the benefit of the doubt. They get blocked.

Cloudflare's July 1 options post already named the blast radius. Googlebot, Applebot, and BingBot sit in that mixed bucket. Block Training and the most restrictive rule wins. Search stays allowed on paper while the mixed crawler still eats a block.

That is not a thought experiment. In August 2026, pete.batin on the Cloudflare community blocked Training on all pages, then watched Google Search Console 403 the homepage and drop it from the index. The old Block AI Bots toggle had left Google alone. The new Training block did not. Ads-only or Allow was the way back.

September 15 2026, new domains default Training and Agent to blocked on pages with ads. Search stays allowed. Mixed-purpose crawlers follow the Training block. Opt out before that date if you like being crawled more than you like the default.

Open the dashboard. Open /robots.txt. If they disagree, a crawler can still point at the gap. You'll ship Sync, feel tidy, and get crawled anyway if the edge is still on Allow.

The file is a guestbook. Write it like one. Lock the door with the thing that drops packets.

The docs-traffic panic around what Google's AI Overviews take from developer documentation is a different job. Discoverability is a strategy. A Disallow is a sentence. Do not confuse the two just because both mention robots.txt.