"Cloudflare has turned its attention from erecting bot barriers to dangling bot bait. Having previously devised a mechanism to make AI crawlers pay to consume website content, the content delivery network is now offering web publishers a way to make it cheaper for AI services to harvest site content by converting HTML to Markdown, the minimalist markup language for representing text mixed with formatting characters in a way that retains legibility."
""Feeding raw HTML to an AI is like paying by the word to read packaging instead of the letter inside," explain Martinho and Allen. "A simple ## About Us on a page in markdown costs roughly 3 tokens; its HTML equivalent - <h2 class="section-title" id="about">About Us</h2> - burns 12-15, and that's before you account for the <div> wrappers, nav bars, and script tags that pad every real web page and have zero semantic value.""
Cloudflare now offers an option for publishers to serve Markdown-formatted responses to AI crawlers, converting HTML into a minimalist, semantically focused representation. Markdown reduces the number of formatting characters and identifiers that AI models must process, lowering token counts and computational cost. AI crawlers can request Markdown via the Accept header during network negotiation. When enabled by a publisher, Cloudflare responds with Markdown and includes an x-markdown-tokens header reporting token count to help services gauge ingestion costs. The feature targets growing AI and agent traffic and aims to streamline content harvesting while making token accounting transparent.
Read at Theregister
Unable to calculate read time
Collection
[
|
...
]