AI Crawlability audit
The AI Crawlability audit checks whether AI bots (GPTBot, ClaudeBot, PerplexityBot, and others) can access and render your website — a prerequisite for being cited in AI responses.
Opening the audit
Section titled “Opening the audit”Click AI Crawlability audit under Tools in the sidebar.
Pages tab
Section titled “Pages tab”Configure which URLs to include in each audit run:
- Discover pages — opens a dialog to load URLs from your sitemap or run automatic discovery
- Manage pages — view and remove active audit pages in a separate dialog (keeps the main view compact)
- Manual add — in the discover dialog, enter a URL, optionally Check URL for HTTP 200, or confirm the URL is correct when your CDN blocks our server
Results tab
Section titled “Results tab”Run an audit and review results:
AI Readiness Score
Section titled “AI Readiness Score”An overall score (0–100) summarising how well your site is configured for AI crawlers. Category scores cover bot access, structured data, content, technical setup, renderability, and discoverability.
Per-page results
Section titled “Per-page results”Each audited page shows its individual score and final URL after redirects.
Recommendations
Section titled “Recommendations”Rule-based, prioritized fixes from failing and warning checks — not generated by AI.
Guided fixes
Section titled “Guided fixes”llms.txt draft — generate a starter llms.txt from your brand settings and configured audit pages. The draft only lists pages in your audit configuration; add more under Manage pages if needed, then regenerate. Copy or download and publish at /llms.txt on your domain.
Schema guidance — copyable JSON-LD snippets (Organization, WebPage, FAQPage) based on structured-data gaps from your latest audit.
AI summary (optional)
Section titled “AI summary (optional)”When Vertex AI is configured, a short executive narrative summarizes audit results for stakeholders. Scores and recommendations remain deterministic — the summary does not invent new issues.
Audit history
Section titled “Audit history”Previous audit runs so you can track improvements after fixing issues.
What gets checked
Section titled “What gets checked”| Category | Examples |
|---|---|
| AI bot access | robots.txt rules for GPTBot, ClaudeBot, PerplexityBot |
| Renderability | JavaScript rendering, content visible without JS, semantic HTML landmarks |
| Structured data | Schema.org markup, JSON-LD |
| Content | Title length (10–70), meta description (25–160), single H1, heading hierarchy, image alt text, visible text, author/date signals |
| Technical | HTTP status, redirect chains, canonical tags |
| Discoverability | llms.txt presence, sitemap, robots.txt |