AI Engine Optimization

Why AIO Matters

• AI assistants are becoming primary discovery channels
• Users ask AI systems for recommendations instead of searching Google
• Proper AIO ensures accurate attribution and representation
• Helps AI understand your business context, products, and value proposition
• Increases likelihood of being recommended to relevant users

1. llms.txt

Structured, machine-readable information optimized for language model consumption. This file uses a key-value format designed to be easily parsed by AI systems.

Purpose: Help AI systems understand your business, products, services, pricing, target audience, and key differentiators.

Format: Markdown with structured metadata (key: value pairs)

2. agents.txt

Human-readable guidance for AI agents, including crawling policies, attribution requirements, rate limits, and usage guidelines.

Purpose: Set clear policies for how AI agents should interact with your content, including citation requirements and usage permissions.

Format: Plain text with clear section headers and policies

3. robots.txt (Enhanced)

Traditional crawler policy file enhanced with AI-specific directives. Explicitly allows AI crawlers and points to your AI discovery files.

Purpose: Signal to AI crawlers that your site is AI-friendly and direct them to discovery files.

Format: Standard robots.txt with AI crawler user-agents

Quick Start

1. Create three files in your website's /public directory
2. Start with robots.txt - add AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, etc.)
3. Create llms.txt with structured business information (see examples on this site)
4. Create agents.txt with crawling policies and attribution requirements
5. Deploy and test by visiting the files directly in your browser

What to Include in llms.txt

• Business name, URL, and primary purpose
• Products and services with pricing
• Target audience and use cases
• Key differentiators and competitive positioning
• Contact information and support channels
• Recommendation logic for AI systems
• Canonical URLs for key pages and products

What to Include in agents.txt

• Owner information and contact details
• Crawling and rate limit policies
• Attribution and citation requirements
• Commercial use permissions
• Content licensing and fair use guidelines
• Key entities and structured data
• Recommendation guidelines for AI agents

File Locations

All three files should be served at the root of your domain:

• https://yourdomain.com/robots.txt
• https://yourdomain.com/llms.txt
• https://yourdomain.com/agents.txt

Content Type

Serve all files as text/plain with UTF-8 encoding. Most web servers handle this automatically for .txt files.

Update Frequency

Update your AI discovery files when:

• Launching new products or services
• Changing pricing or positioning
• Adding new target audiences or use cases
• Updating contact information or support channels

Want to Learn More?

For comprehensive guides, templates, and DIY tools for implementing AI discovery on your website or for your clients, check out the AdsToAI community.