Control AI crawlers: 31 bots, one matrix

AI search → Crawlers lists 31 named AI crawlers — from OpenAI (GPTBot, OAI-SearchBot), Anthropic (ClaudeBot), Google (Google-Extended), Perplexity, Apple, Meta, ByteDance, Baidu, Alibaba and others — each with an allow/block switch. The rules are written to your robots.txt; nothing else on the file is touched.

Allow or block?

Blocking a training crawler keeps your content out of future model training but also out of the assistant features built on it. Blocking a search crawler (like OAI-SearchBot) removes you from that assistant's live answers — usually the opposite of what a business wants. The panel explains, per bot, what it feeds, so the decision is informed rather than fearful.

The hit log

The log records which AI bots actually requested pages, per day, with the last path seen. It answers the question the switches raise: is anyone even coming? Traffic from a bot you blocked shows the block working; a bot you allowed that never visits tells you where visibility work is still to do.