davidrevoy.com
robots.txt

Robots Exclusion Standard data for davidrevoy.com

Resource Scan

Scan Details

Site Domain davidrevoy.com
Base Domain davidrevoy.com
Scan Status Ok
Last Scan2025-12-16T14:06:10+00:00
Next Scan 2026-01-15T14:06:10+00:00

Last Scan

Scanned2025-12-16T14:06:10+00:00
URL https://davidrevoy.com/robots.txt
Redirect https://www.davidrevoy.com/robots.txt
Redirect Domain www.davidrevoy.com
Redirect Base davidrevoy.com
Domain IPs 213.186.33.87
Redirect IPs 213.186.33.87
Response IP 213.186.33.87
Found Yes
Hash fc8beeff5fa49f8b53f2bdafe7659b48a6d6a6ad7f132ca1e7f04b967b8b3eab
SimHash 701d0941c0e4

Groups

*

Rule Path
Disallow /tmp/
Disallow /tmp/cache/
Disallow /downloader.php
Disallow /themes/peppercarrot-theme_v2/cat-avatar-generator.php
Disallow /plugins/vignette/plxthumbnailer.php

Comments

  • AI Crawler list, src: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
  • https://github.com/ai-robots-txt/ai.robots.txt , MIT license.
  • User-agent: AI2Bot
  • User-agent: Ai2Bot-Dolma
  • User-agent: Amazonbot
  • User-agent: anthropic-ai
  • User-agent: Applebot
  • User-agent: Applebot-Extended
  • User-agent: Brightbot 1.0
  • User-agent: Bytespider
  • User-agent: CCBot
  • User-agent: ChatGPT-User
  • User-agent: Claude-Web
  • User-agent: ClaudeBot
  • User-agent: cohere-ai
  • User-agent: cohere-training-data-crawler
  • User-agent: Crawlspace
  • User-agent: Diffbot
  • User-agent: DuckAssistBot
  • User-agent: FacebookBot
  • User-agent: FriendlyCrawler
  • User-agent: Google-Extended
  • User-agent: GoogleOther
  • User-agent: GoogleOther-Image
  • User-agent: GoogleOther-Video
  • User-agent: GPTBot
  • User-agent: iaskspider/2.0
  • User-agent: ICC-Crawler
  • User-agent: ImagesiftBot
  • User-agent: img2dataset
  • User-agent: ISSCyberRiskCrawler
  • User-agent: Kangaroo Bot
  • User-agent: Meta-ExternalAgent
  • User-agent: Meta-ExternalFetcher
  • User-agent: OAI-SearchBot
  • User-agent: omgili
  • User-agent: omgilibot
  • User-agent: PanguBot
  • User-agent: PerplexityBot
  • User-agent: PetalBot
  • User-agent: Scrapy
  • User-agent: SemrushBot-OCOB
  • User-agent: SemrushBot-SWA
  • User-agent: Sidetrade indexer bot
  • User-agent: Timpibot
  • User-agent: VelenPublicWebCrawler
  • User-agent: Webzio-Extended
  • User-agent: YouBot
  • Disallow: /