Cloudflare, a prominent internet infrastructure provider, is taking a significant step to protect content creators. They are implementing a default setting to block AI crawlers from accessing website content without explicit permission or compensation. This move aims to give content creators more control over how their work is utilized by AI companies.
Starting July 1, 2025, new web domains using Cloudflare's services will be prompted to decide on AI crawler access. This shifts the responsibility to AI companies to request access and clarify their crawler's purpose. These purposes include training, inference, or search functions.
Cloudflare's initiative addresses concerns about AI crawlers scraping content without limits. This practice has deprived content creators of revenue and recognition. By blocking unauthorized scraping, Cloudflare seeks to safeguard the internet's future.
In addition to blocking crawlers, Cloudflare has introduced a "Pay Per Crawl" program. This feature allows content creators to set a price for AI companies using their content. This is for model training and retrieval-augmented generation. The program is currently in private beta.
Major publishers, including Gannett and The Associated Press, support Cloudflare's initiative. They see this as a step towards a fair value exchange. This protects creators and supports quality journalism.
Cloudflare's move represents a significant change for content creators in the AI age. By providing tools to block and charge for content access, Cloudflare sets a new standard for respecting intellectual property online.