How to Make Your Website AI-Agent Friendly

AI agents and LLMs waste tokens parsing raw HTML. By adding an llms.txt file and serving markdown versions of your pages, you make your site easy for AI to understand.

What are these files?

llms.txtDiscovery

A file at your site root that tells AI agents what your site is about and where to find key pages. Think of it like robots.txt, but for LLMs.

.md filesContent

Markdown versions of your pages that AI agents can consume directly. Up to 10x smaller than HTML, with no noise from nav bars, ads, or scripts.

Part 1: Adding llms.txt

The llmstxt.org spec defines a simple format that gives AI agents an overview of your site.

1Generate your llms.txt

Use the converter to convert any page on your site, then go to the llms.txt tab to generate the file. Or create it manually following this format:

llms.txt

# Your Site Name

> A brief description of what your site/product does.

## Docs

- [Getting Started](https://yoursite.com/docs/getting-started): How to set up and install
- [API Reference](https://yoursite.com/docs/api): Full API documentation
- [Pricing](https://yoursite.com/pricing): Plans and pricing details

## Blog

- [Announcing v2.0](https://yoursite.com/blog/v2-launch): Major release with new features

2Place it at your site root

The file must be accessible at https://yoursite.com/llms.txt.

Next.js / React:

public/llms.txt

# Your Site Name

> Description of your site.

## Docs

- [Getting Started](https://yoursite.com/docs/getting-started)

WordPress:

Upload llms.txt to your WordPress root directory via FTP, or use a plugin like "LLMs.txt" from the plugin directory.

Static sites (Hugo, Jekyll, etc.):

Place llms.txt in your static/ or public/ folder.

3Verify it works

Visit https://yoursite.com/llms.txt in your browser. You should see the plain text content.

Part 2: Adding Markdown Versions of Pages

For each key page, you can serve a markdown version that AI agents can consume directly. This is especially useful for documentation, blog posts, and landing pages.

1Convert your pages to markdown

Use the converter to paste each URL and download the .md file.

2Choose a URL scheme

There are two common approaches:

Option A: Same URL with .md extension

/docs/getting-started → /docs/getting-started.md

Option B: llms-full.txt for entire site

Serve all your markdown content in a single file at /llms-full.txt

3Serve the markdown files

Next.js (App Router):

app/docs/[slug]/md/route.ts

import { NextRequest } from "next/server";
import fs from "fs";
import path from "path";

export async function GET(
  request: NextRequest,
  { params }: { params: { slug: string } }
) {
  const filePath = path.join(
    process.cwd(), "content", `${params.slug}.md`
  );
  const content = fs.readFileSync(filePath, "utf-8");

  return new Response(content, {
    headers: { "Content-Type": "text/markdown; charset=utf-8" },
  });
}

Static sites:

Simply place the .md files alongside your HTML pages in the build output directory.

Nginx / Apache:

nginx.conf

location ~ \.md$ {
    default_type text/markdown;
    root /var/www/yoursite/markdown;
}

4Link from llms.txt

Update your llms.txt to point to the markdown versions:

llms.txt

# Your Site Name

> Description of your site.

## Docs

- [Getting Started](https://yoursite.com/docs/getting-started.md)
- [API Reference](https://yoursite.com/docs/api.md)

Best Practices

Keep llms.txt concise

Only include your most important pages. AI agents don't need every blog post — focus on docs, guides, and key landing pages.

Update when content changes

Regenerate your markdown files when the source pages change. Stale content is worse than no content.

Include descriptions

Add notes after each link in llms.txt so agents know what they'll find: "- [API Docs](url): REST API endpoints and authentication"

Test with an LLM

Paste your llms.txt or markdown into ChatGPT or Claude and ask it questions about your product. If it answers well, your files are working.

Set correct content types

Serve .md files with Content-Type: text/markdown and llms.txt with text/plain.

Ready to get started?

Convert your first page and generate your llms.txt file in seconds.

Convert a page