$ cat /var/log/thoughts

Cloudflare + Security + Web Dev

Posts matching all selected tags.

Bot Preference Sync admits robots.txt never enforced

Bot Preference Sync writes your Search, Agent, and Training policy into robots.txt. A mismatch is an argument to keep crawling. The WAF is still the lock.

Strong OpinionsCloudflareSecurityWeb DevDevOpsTrend Commentary1 min read · Aug 22, 2026

Web Scraping in the AI Age: What Actually Changed

Web scraping in the AI age broke twice over: LLMs killed the CSS selector, and AI crawlers exploded while referral traffic collapsed. Here is what changed.

Strong OpinionsWeb DevAI CodingSecurityAutomationPythonCloudflareTrend Commentary1 min read · May 31, 2026

How to Block AI Crawlers From Destroying Your Site

AI crawlers from OpenAI, Meta, and Anthropic are hammering sites with millions of requests per month. robots.txt alone does not stop them. Here are the 4 layers that actually work.

Build AlongSecurityDevOpsWeb DevNginxCloudflare1 min read · Apr 11, 2026

Cloudflare Is the Biggest Man-in-the-Middle in History (And You're OK With It)

Cloudflare terminates TLS for roughly 20% of all web traffic. Every proxied HTTPS request is decrypted and re-encrypted on their servers. Here's what that means, why we accept it, and what the alternatives actually look like.

Strong OpinionsSecurityCloudflareEncryptionNetworkingWeb DevTrend Commentary1 min read · Apr 11, 2026