$ cat /var/log/thoughts
Cloudflare + DevOps + Security
Posts matching all selected tags.
Category
All Categories
Tags
Newsletter
New posts land in your inbox when they publish. No spam, unsubscribe anytime.
You're in. New posts head your way.
Filters3
2 posts
Bot Preference Sync admits robots.txt never enforced
Bot Preference Sync writes your Search, Agent, and Training policy into robots.txt. A mismatch is an argument to keep crawling. The WAF is still the lock.
Strong OpinionsCloudflareSecurityWeb DevDevOpsTrend Commentary1 min read · Aug 22, 2026
How to Block AI Crawlers From Destroying Your Site
AI crawlers from OpenAI, Meta, and Anthropic are hammering sites with millions of requests per month. robots.txt alone does not stop them. Here are the 4 layers that actually work.
Build AlongSecurityDevOpsWeb DevNginxCloudflare1 min read · Apr 11, 2026