"We probably didn't get this launch right last week and should apologize for that. I think our messaging wasn't very good on it. We should have led with the message that it respects the existing controls, and the intention is to provide a crawler that respects publishers' wishes."
"A ripple effect of that speed of development has meant that the publisher user experience means that clients have six different surfaces where they can express their bot blocking preferences, resulting in a user experience that isn't where we want it to be."
Cloudflare released a crawl API within its browser rendering service that enables scraping entire websites with a single request, returning content in HTML, Markdown, or JSON formats. The launch generated significant controversy, with publishers expressing concern about the irony of Cloudflare—historically protective of publisher interests against scrapers—now providing its own scraping tool. Initial implementation issues prevented publishers from blocking Cloudflare's crawler using standard settings, prompting vocal criticism from independent publishers. Cloudflare's leadership acknowledged messaging failures and technical oversights in the rollout, attributing problems to rapid development cycles and fragmented product ownership. The company emphasized its intention to create a compliant crawler respecting publisher preferences while addressing user experience inconsistencies across multiple bot-blocking interfaces.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]