Feature Spotlight: AI Indexer

Your Content. Your Rules.
The AI Firewall.

Stop giving your data away for free to training models. LLM Optimizer gives you granular control over who enters your site—blocking aggressive scrapers while rolling out the red carpet for traffic-driving assistants.

The Threat

The “Black Box” of AI Scraping

Right now, dozens of AI bots are crawling your site. Some want to send you traffic (Perplexity, Bing). Others just want to steal your content to train their models (GPTBot, CCBot) without ever giving you a click.

Without LLM Optimizer:
  • Resource Drain: Aggressive scraping slows down your server for real humans.
  • Data Theft: Your hard work trains their models for free.
  • Zero Visibility: You have no idea which pages are being indexed by AI.

Granular Provider Rules

We don’t use a blanket “Block All” strategy. We identify bots by their User-Agent and IP ranges, allowing you to set specific policies for each provider.

Allow Traffic Sources

Automatically whitelist bots that drive click-through traffic, such as Perplexity, Bing Chat, and Google Gemini. Ensure you appear in their citations.

Block Data Scrapers

Detect and block “Training Only” bots like Common Crawl (CCBot) and Anthropic-Scraper. Keep your unique data proprietary.

Spoof Protection

Hackers often fake their User-Agent to look like Googlebot. We verify bot IP addresses against official CIDR ranges to ensure the visitor is who they say they are.

The Control Center

Manage your AI visibility with simple toggles.

Set Your Policy

You decide the relationship you want with AI.

  • Maximum Visibility: Allow all bots to ingest content for maximum brand awareness.
  • Balanced: Allow search assistants, block model trainers.
  • Dark Mode: Block all AI scrapers completely.
Take Control Now
LLM Optimizer > Provider Rules
OpenAI (ChatGPT)

User Agents: GPTBot, ChatGPT-User

ALLOWED

Anthropic (Claude)

User Agents: ClaudeBot

ALLOWED

Common Crawl

User Agents: CCBot (Training Data)

BLOCKED