Bot Library

Every bot that quietly visits your site — in one place

84 crawlers across search engines, AI assistants, social previews, SEO tools, and archives. We tell you who they are, what they do, and which ones RenderBeam can serve a fully-rendered page to on your plan.

How search and AI bots actually work

Every bot below behaves a little differently. Understanding the mechanics is the difference between getting indexed and quietly disappearing.

They fetch raw HTML — fast

Most bots make a single HTTP request and read what comes back. They don't wait for your JavaScript to render. If your content isn't in the initial HTML, it doesn't exist to them.

They have a crawl budget

Bots only spend so much time on each site. Slow responses, JS-rendered pages, and dead links burn that budget — meaning fewer of your pages get indexed.

They identify themselves

Every bot sends a User-Agent string like 'GPTBot/1.0' or 'Googlebot/2.1'. RenderBeam matches that string to serve them a clean, rendered page — and shows you exactly who hit you.

What's available on each plan

Trial unlocks 24 top crawlers. Every paid plan unlocks all 84.

The Trial preset focuses on the bots most sites actually need (Google, Bing, OpenAI, Perplexity, Anthropic, Facebook, LinkedIn, etc). Upgrade any time to unlock the full library — same flat price, no per-bot fees.

Search engines

16 of 16

Classic crawlers that build the indexes powering Google, Bing, DuckDuckGo, and regional search.

Paid

360 Search

Search engines crawler.

Trial

Apple

Search engines crawler.

Paid

Baidu

Search engines crawler.

Trial

Bing

Search engines crawler.

Trial

DuckDuckGo

Search engines crawler.

Paid

Ecosia

Search engines crawler.

Trial

Google

Search engines crawler.

Paid

Mojeek

Search engines crawler.

Paid

Naver

Search engines crawler.

Paid

Petal (Huawei)

Search engines crawler.

Paid

Qwant

Search engines crawler.

Paid

Seznam

Search engines crawler.

Paid

Shenma

Search engines crawler.

Paid

Sogou

Search engines crawler.

Trial

Yahoo

Search engines crawler.

Paid

Yandex

Search engines crawler.

AI crawlers

24 of 24

Bots that feed answers in ChatGPT, Claude, Perplexity, Grok, Gemini, and other AI assistants.

Paid

Amazonbot

AI crawlers crawler.

Paid

Andi

Andi AI search crawler

Trial

Anthropic

AI crawlers crawler.

Paid

Apple Intelligence

Apple Intelligence training opt-out token

Paid

Brave Search

Brave Search crawler

Paid

Bytespider

ByteDance AI training crawler

Paid

Cohere

Cohere AI training crawler

Paid

CommonCrawl

AI crawlers crawler.

Paid

DeepSeek

DeepSeek AI crawler

Paid

Diffbot

Diffbot knowledge graph + AI crawler

Paid

DuckAssistBot

AI crawlers crawler.

Trial

Google Extended

AI crawlers crawler.

Paid

Google Vertex

Google Cloud Vertex AI agent

Paid

Grok (xAI)

xAI Grok crawler

Paid

ImageSift

ImageSift AI image crawler

Paid

Kagi

Kagi search/AI crawler

Paid

Meta AI

Meta AI on-demand fetcher (separate from Facebook link preview)

Paid

Mistral

AI crawlers crawler.

Trial

OpenAI

AI crawlers crawler.

Trial

Perplexity

AI crawlers crawler.

Paid

PetalBot AI

Huawei Aspiegel AI crawler

Paid

Phind

Phind AI search crawler

Paid

Timpi

Timpi decentralized search crawler

Paid

You.com

AI crawlers crawler.

Social previews

21 of 21

Bots that generate the link cards you see in Slack, Discord, Twitter, LinkedIn, and messengers.

Trial

Bitly

Bitly URL shortener preview

Paid

Discord

Social previews crawler.

Paid

Embedly

Embedly link preview service

Trial

Facebook

Social previews crawler.

Paid

Flipboard

Flipboard content aggregator

Trial

Instagram

Social previews crawler.

Trial

Social previews crawler.

Paid

Nuzzel

Nuzzel news aggregator

Trial

Outbrain

Outbrain content recommendation

Trial

Social previews crawler.

Paid

Quora

Quora link preview bot

Trial

Social previews crawler.

Paid

Skype

Social previews crawler.

Trial

Slack

Social previews crawler.

Trial

Social previews crawler.

Trial

TikTok

Social previews crawler.

Paid

Tumblr

Tumblr blog platform crawler

Paid

Twitter/X

Social previews crawler.

Paid

VK/VKontakte

VK social network share bot

Trial

Social previews crawler.

Paid

XING

XING professional network bot

SEO tools

18 of 18

Crawlers from SEO suites like Ahrefs, Semrush, Moz, Screaming Frog — useful for audits.

Trial

Ahrefs

SEO tools crawler.

Paid

Botify

SEO tools crawler.

Paid

Chrome Lighthouse

SEO tools crawler.

Trial

Google PageSpeed

Google PageSpeed Insights crawler

Paid

Google PSI/Inspection

SEO tools crawler.

Paid

Lumar (Deepcrawl)

SEO tools crawler.

Paid

Majestic

SEO tools crawler.

Paid

Moz DotBot

SEO tools crawler.

Paid

Moz Rogerbot

SEO tools crawler.

Paid

OnCrawl

SEO tools crawler.

Paid

OpenLinkProfiler

SEO tools crawler.

Paid

ScreamingFrog

SEO tools crawler.

Paid

SEMrush

SEO tools crawler.

Paid

Seobility

SEO tools crawler.

Paid

Serpstat

SEO tools crawler.

Paid

Sitebulb

SEO tools crawler.

Paid

Ubersuggest

SEO tools crawler.

Trial

W3C Validator

W3C HTML/CSS validation service

Archives & misc

5 of 5

Long-term web archives and a few legacy or special-purpose fetchers.

Paid

Bitrix

Bitrix24 link preview service

Trial

Google Snippet

Google+ snippet fetcher (legacy)

Paid

Integration Test

Testing/development bot identifier

Paid

Internet Archive / Alexa

Archives & misc crawler.

Paid

ShowYouBot

ShowYou video discovery bot

Stop guessing who crawls your site

RenderBeam shows every bot that hits your domain in real time — Googlebot, GPTBot, ClaudeBot, Perplexity, and 80+ more. See what they read, what they 404 on, and serve every single one a clean, fully-rendered page.