Log In Sign Up

fujifilm-x.com
robots.txt

Robots Exclusion Standard data for fujifilm-x.com

Resource Scan

Scan Details

Site Domain fujifilm-x.com
Base Domain fujifilm-x.com
Scan Status Ok
Last Scan2024-10-29T05:32:22+00:00
Next Scan 2024-11-28T05:32:22+00:00

Last Scan

Scanned2024-10-29T05:32:22+00:00
URL https://fujifilm-x.com/robots.txt
Domain IPs 141.193.213.20, 141.193.213.21
Response IP 141.193.213.21
Found Yes
Hash f49a548b6ca8fb0f8704f8c917ef78a8c47a5a0d31da90e632d61d9c41b8b8e2
SimHash 769a5960c4a5

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 600

meta-externalagent

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /
Back to top

Other Records

Field Value Comment
sitemap https://fujifilm-x.com/global/sitemap_index.xml Sitemap for English (and all other sites not listed)
sitemap https://fujifilm-x.com/en-us/sitemap_index.xml Sitemap for English (US)
sitemap https://fujifilm-x.com/en-ca/sitemap_index.xml Sitemap for English (Canada)
sitemap https://fujifilm-x.com/fr-ca/sitemap_index.xml Sitemap for French (Canada)
sitemap https://fujifilm-x.com/en-gb/sitemap_index.xml Sitemap for English (Great Britain)
Back to top

Comments

  • Allow all search engines to crawl the entire site, except for specific directories and files
  • Disable the meta AI crawler
  • Disallow GPTBot (AI-based web crawler by OpenAI) from crawling the site
  • Disallow Google-Extended (Google services beyond web search, such as AI training) from crawling the site
  • Disallow CCBot (Common Crawl's web crawler) from crawling the site
  • Disallow Claude-Web (Web crawler by Anthropic, for AI and research purposes) from crawling the site
  • Disallow anthropic-ai (Anthropic's AI services crawler) from crawling the site
  • Disallow YandexBot (Web crawler by Yandex, a Russian search engine) from crawling the entire site
  • Blocks the BLEXBot which was causing 10x increases in throughput.
  • Provide the location of the sitemap for each language version (replace with your actual sitemap URLs)
  • Search engines will use these sitemaps to discover the pages on your website.
Back to top