builtinvacuum.com
robots.txt

Robots Exclusion Standard data for builtinvacuum.com

Resource Scan

Scan Details

Site Domain builtinvacuum.com
Base Domain builtinvacuum.com
Scan Status Ok
Last Scan2024-10-26T05:13:31+00:00
Next Scan 2024-11-25T05:13:31+00:00

Last Scan

Scanned2024-10-26T05:13:31+00:00
URL https://builtinvacuum.com/robots.txt
Domain IPs 206.190.69.19
Response IP 206.190.69.19
Found Yes
Hash e0343719f81e6964606ea53fa89e3b95ded9982f62b92394951e2157da71d5f6
SimHash 58155b51b5d0

Groups

*

Rule Path
Disallow /dealer/
Disallow /Scripts/AddToCart.php
Disallow /*.pdf$

ia_archiver

Rule Path
Disallow /

updownerbot

Rule Path
Disallow /

googlebot

Rule Path
Allow .js

scrapy
magpie-crawler
ccbot
omgili
omgilibot
node/simplecrawler

Rule Path
Disallow /

gptbot
chatgpt-user
claude-web
claudebot
anthropic-ai
cohere-ai
bytespider
perplexitybot
applebot-extended
diffbot
perplexitybot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://builtinvacuum.com/sitemap.xml

Comments

  • scrapers
  • ai chatbots
  • User-agent: Google-Extended