Here's a question most website owners haven't considered: when an AI agent visits your site, what does it actually see?

Not what a human sees — a clean layout, styled headings, a navigation bar. What the machine receives: a wall of HTML containing navigation menus, cookie consent banners, SVG icons, JavaScript bundles, footer links, tracking scripts, and somewhere in that noise, the actual content the agent came for. On a typical page, 70–80% of the tokens an AI model processes are structural markup that carries zero informational value.

This is the web's newest accessibility problem. And unlike screen reader compatibility, which the industry took decades to address properly, this one is moving fast.

What Happened

In February 2026, Cloudflare introduced a feature called "Markdown for Agents." The concept is elegant: when an AI agent sends a request with Accept: text/markdown in its HTTP headers, the server responds with clean markdown instead of HTML. Same URL, same content, different representation — selected through standard HTTP content negotiation.

It's a sensible move. Cloudflare sits in front of roughly 20% of the web's traffic, and they've been watching AI crawlers consume bandwidth at scale. Giving those crawlers a lightweight, structured format reduces load on everyone — the origin server, the CDN, the AI service processing the response, and ultimately the user waiting for an answer.

The feature works by converting HTML to markdown at the edge, stripping out navigation, scripts, styles, and other non-content elements. The result is a clean document that AI models can process efficiently — fewer tokens, less noise, better comprehension.

There's just one catch: you have to use Cloudflare.

Content Negotiation Is Not a Cloudflare Invention

What Cloudflare built is clever engineering on top of a mechanism that has existed since HTTP/1.1 was standardised in 1997. Content negotiation — the process by which a client specifies what format it prefers and the server responds accordingly — is defined in RFC 9110, Section 12.

The Accept header is how your browser already tells servers it wants HTML over plain text, or how API clients request JSON instead of XML. When an AI agent sends Accept: text/markdown, it's using the exact same mechanism. Any web server can respond to it. You don't need Cloudflare. You don't need any specific CDN. You need a web server that can read a header and serve a different file.

We decided to implement this on our own infrastructure — without Cloudflare, without any third-party dependency — because (a) we wanted to understand the full pipeline, and (b) our clients don't all use the same CDN, and they shouldn't have to.

How We Built It

Our site runs on nginx serving static HTML, with a FastAPI backend handling the blog, search, and admin functions. The architecture needed to handle two fundamentally different content types: static pages that rarely change, and blog posts that are stored in a database and updated regularly.

The nginx Layer

The entry point is a map directive in the nginx configuration that inspects the Accept header:

map $http_accept $wants_markdown {
    default         0;
    "~text/markdown" 1;
}

This creates a variable that downstream location blocks can use to route requests. When a regular browser visits /en/about.html, it gets HTML as usual. When an AI agent requests the same URL with Accept: text/markdown, the request is rewritten to our markdown API:

location /en/ {
    if ($wants_markdown) {
        rewrite ^/en/(.*)\.html$ /api/markdown/page/en/$1 last;
        rewrite ^/en/$ /api/markdown/page/en/index last;
    }
    try_files $uri $uri/ /en/404.html;
}

The agent never sees the rewrite. Same URL, different response — precisely how content negotiation is supposed to work.

The Hybrid Conversion Strategy

Converting HTML to markdown in real time for every request would be wasteful. Our static pages — about, services, products, case studies — change at most a few times per month. Converting them on every agent visit would burn CPU cycles for identical output.

So we split the approach:

• Static pages are pre-generated to markdown files on disk. A nightly scheduled job walks the HTML source directory, strips non-content elements, converts the remainder to clean markdown with YAML frontmatter, and writes the results to a known path. The API endpoint simply serves these files — no processing, no conversion, just a file read.
• Blog posts are converted in real time from the database. Since posts are already stored as structured HTML content (no navigation, no chrome — just the article body), the conversion is lightweight: parse the HTML, convert to markdown, prepend metadata as frontmatter, and serve.

The pre-generation runs at 02:30 UTC daily and can be triggered manually through an admin endpoint. Currently it processes 67 pages and generates approximately 86,000 tokens of clean markdown.

What Gets Stripped

The conversion pipeline removes everything that isn't content. Specifically:

• Navigation bars (<nav>, <header>)
• Footers
• Scripts and stylesheets
• SVG elements and inline icons
• Cookie consent banners
• Form elements
• Skip-to-content links and accessibility overlays
• Any element with classes matching common non-content patterns (cookie, consent, newsletter, popup)

What remains is the semantic content: headings, paragraphs, lists, links, code blocks, images with alt text, and tables. This is what the AI agent actually needs.

The Frontmatter

Each markdown response begins with YAML frontmatter containing structured metadata:

---
title: "IT Infrastructure Services"
description: "Enterprise IT architecture, governance, and optimisation"
url: https://iwh.gr/en/services/infrastructure.html
language: en
type: article
generated: "2026-02-22T02:30:14Z"
tokens: 1847
---

For blog posts, the frontmatter is richer — including author, publication date, category, and tags. This structured metadata is immediately useful to AI models without requiring them to parse it from HTML meta tags buried in the <head>.

What AI Agents Actually See

The difference is dramatic. Here's what a typical page looks like to an AI agent requesting HTML versus markdown.

HTML response (simplified — the real thing is worse):

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>About IWH</title>
  <link rel="stylesheet" href="/assets/css/style.min.css">
  <script src="/assets/js/main.min.js"></script>
  <!-- 40 more meta tags, OG tags, schema markup... -->
</head>
<body>
  <header>
    <nav>
      <!-- 200+ lines of navigation HTML -->
    </nav>
  </header>
  <main>
    <h1>About IWH</h1>
    <p>The actual content you came for.</p>
    <!-- 20-30 lines of real content -->
  </main>
  <footer>
    <!-- Another 100+ lines -->
  </footer>
  <div class="cookie-consent">...</div>
  <script>/* analytics, consent, animations */</script>
</body>
</html>

That's roughly 15,000–25,000 characters for a page with 2,000 characters of actual content. The signal-to-noise ratio is terrible.

Markdown response for the same URL:

---
title: "About IWH"
description: "IT infrastructure, cybersecurity, and compliance advisory"
url: https://iwh.gr/en/about.html
language: en
tokens: 487
---

# About IWH

The actual content you came for.

## Our Background

IWH — Internet Why's and How's — was founded to address
a persistent gap in IT and security consulting...

Clean, structured, immediately parseable. No navigation to skip, no scripts to ignore, no cookie banners to filter out. The token count drops by 80–90%, and what remains is pure content.

The Response Headers

Beyond the content itself, we include headers that signal intent to AI systems:

Content-Type: text/markdown; charset=utf-8
Vary: Accept
X-Content-Signal: ai-train=yes, search=yes, ai-input=yes
X-Markdown-Tokens: 487
X-Robots-Tag: noarchive

The Vary: Accept header is critical — it tells caches and CDNs that the response varies based on the Accept header, preventing a cached markdown response from being served to a browser (or vice versa). The X-Content-Signal header is an emerging convention for explicitly granting AI systems permission to use the content. The token count in X-Markdown-Tokens lets agents estimate processing costs before reading the body.

The Root URL Problem

There's an interesting edge case with the root URL. When you visit https://iwh.gr/ in a browser, you get redirected to /en/. But an AI agent requesting the root with Accept: text/markdown doesn't want a redirect — it wants a machine-readable overview of the entire site.

We handle this by serving our llms-full.txt file to markdown-requesting agents at the root:

location = / {
    if ($wants_markdown) {
        rewrite ^ /llms-full.txt last;
    }
    return 301 /en/;
}

This file — following the emerging llms.txt convention — provides a structured overview of the entire site: what we do, what pages exist, what content is available. It's the machine-readable equivalent of a human scanning the homepage and navigation to understand what a company offers.

Beyond Cloudflare: Why Self-Hosted Matters

Cloudflare's implementation is convenient for sites already on their CDN. But the assumption that AI-readability requires a specific vendor is worth challenging:

• CDN independence. Not every organisation uses Cloudflare, and many have contractual or compliance reasons for their CDN choice. Content negotiation should work regardless of infrastructure.
• Customisable conversion rules. Cloudflare's edge conversion is one-size-fits-all. Self-hosted conversion lets you control exactly what gets stripped, what metadata gets included, and how the output is structured for your specific content.
• Data sovereignty. The conversion happens on your infrastructure. Content never passes through a third-party conversion layer.
• Pre-generation efficiency. Cloudflare converts on every request at the edge. Pre-generating static content means zero conversion overhead at request time — the server reads and sends a file, nothing more.
• Blog and dynamic content. Cloudflare converts the full HTML page. Our approach converts only the content from the database, skipping the template entirely. The result is cleaner because the source material is cleaner.

This isn't an argument against Cloudflare — their implementation is well-engineered and appropriate for many sites. It's an argument against assuming that a CDN feature is the only way to solve this problem.

llms.txt: The Complementary Piece

Content negotiation via Accept: text/markdown gives AI agents a clean version of any specific page they request. But it doesn't help an agent that's encountering your site for the first time and needs to understand what's available.

That's where llms.txt comes in. Proposed by Jeremy Howard and gaining traction across the web, llms.txt is a convention (similar to robots.txt or sitemap.xml) where sites provide a machine-readable overview at a known path. Our implementation includes two files:

• /llms.txt — A concise summary: who we are, what we do, and links to key sections
• /llms-full.txt — A comprehensive document covering all pages, services, products, and content in detail

Together with markdown content negotiation, these create a complete AI-readability layer: llms.txt for discovery, markdown responses for consumption. An AI agent can read our llms.txt to understand the site structure, then request specific pages as markdown to get the content it needs — all without parsing a single HTML tag.

Should You Do This?

Honest assessment: it depends.

Probably not worth it if:

• You have a 5-page brochure site with static content that rarely changes. The effort exceeds the benefit.
• You're already on Cloudflare and their implementation covers your needs. Don't reinvent what's working.
• Your content isn't the kind AI agents are likely to consume (purely transactional sites, web apps, dashboards).

Probably worth it if:

• You run a content-heavy site — a blog, knowledge base, documentation portal, or news publication. AI agents are already consuming your content; you should control how.
• You want CDN independence. Your infrastructure choices shouldn't dictate your AI-readability strategy.
• You serve regulated or sensitive content where data sovereignty matters. Content conversion should happen on your infrastructure.
• You care about the emerging AI ecosystem. Sites that are easy for AI agents to read will have an advantage in AI-powered search, citations, and recommendations.

The broader trend is unmistakable. AI agents are becoming a significant source of web traffic, and the sites that make their content cleanly accessible to those agents will benefit disproportionately. Whether you implement this through Cloudflare, self-hosted conversion, or a managed service, the direction is clear: the web needs a machine-readable layer, and content negotiation is the right mechanism to provide it.

The Implementation Checklist

If you decide to implement this, here's the practical sequence:

1. Add llms.txt first. It's the easiest step and immediately makes your site more discoverable to AI systems. Even a simple text file listing your key pages and what they contain is valuable.
2. Implement Accept: text/markdown content negotiation. Start with your most important content pages. Use nginx map + if for routing, and a lightweight conversion service behind it.
3. Pre-generate where possible. Static content should be pre-converted, not computed on every request. Schedule regeneration based on your content update frequency.
4. Handle dynamic content separately. Blog posts, product listings, and frequently updated content should convert in real time from the source data, not from the rendered HTML.
5. Set proper headers. Vary: Accept is non-negotiable. X-Content-Signal for permission signalling. Token counts for cost estimation.
6. Test with real agents. curl -H "Accept: text/markdown" is your friend. Verify every content path returns clean, useful markdown.

---

We built AgentReady to help organisations implement AI-readable content delivery without CDN lock-in. If you're interested in making your website fluent in the language AI agents actually speak, let's talk.