keywords.dev-bot

keywords.dev-bot reads public web pages when someone asks keywords.dev for recommendations about a site or its competitors. It isn't a search engine and doesn't crawl the web in general.

How it behaves

• It follows robots.txt, including rules for the user agent keywords.dev-bot.
• It reads a small sample of pages per site, a few at a time, and caches them for a day.
• It only reads public pages. It never logs in or fills in forms.

Blocking it

Add this to your robots.txt:

User-agent: keywords.dev-bot
Disallow: /

Questions? Email [email protected].