# https://dynamicdata.com/robots.txt # Crawl access policy. For a curated AI content map see https://dynamicdata.com/llms.txt # (llms.txt does not grant or deny access — this file does.) User-agent: * Disallow: /admin/ Disallow: /django-admin/ Disallow: /private/ Disallow: /junk/ # --- AI training opt-out: personal data on Who We Are --- # Blocks training crawlers from /who-we-are/ only. # Regular Googlebot / Bingbot still allowed (SEO). # AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) still allowed site-wide. User-agent: GPTBot Disallow: /who-we-are Disallow: /who-we-are/ User-agent: ClaudeBot Disallow: /who-we-are Disallow: /who-we-are/ User-agent: Google-Extended Disallow: /who-we-are Disallow: /who-we-are/ User-agent: Applebot-Extended Disallow: /who-we-are Disallow: /who-we-are/ User-agent: CCBot Disallow: /who-we-are Disallow: /who-we-are/ User-agent: Bytespider Disallow: /who-we-are Disallow: /who-we-are/ User-agent: meta-externalagent Disallow: /who-we-are Disallow: /who-we-are/ Sitemap: https://dynamicdata.com/sitemap.xml Sitemap: https://dynamicdata.com/feeds/sitemap.xml