Sitemap: https://garrettdimon.com/sitemap.xml

User-agent: *
Disallow: /guide

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: omgili
Disallow: /

## Meta’s bot that crawls public web pages to improve language models for their speech recognition technology.
User-agent: FacebookBot
Disallow: /

## Apple very kindly told us how to block their scraper AFTER they'd scraped everything.
User-agent: Applebot-Extended
Disallow: /

## is used by used by Anthropic to gather data for their “AI” products, such as Claude
User-agent: anthropic-ai
Disallow: /

## is another agent used by Anthropic that is more specifically related to Claude
User-agent: ClaudeBot
Disallow: /

# is a somewhat dishonest scraping bot used to collect data to train LLMs. This is their default user-agent, but they make it easy for their clients to change it to something else and ignore your wishes
User-agent: Diffbot
Disallow: /

## This is just getting stupid and I hope governments step in to wreck these tech-bro thieves.
User-agent: Bytespider
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: cohere-ai
Disallow: /