Log In Sign Up

freedompet.com
robots.txt

Robots Exclusion Standard data for freedompet.com

Resource Scan

Scan Details

Site Domain freedompet.com
Base Domain freedompet.com
Scan Status Ok
Last Scan2024-06-14T08:01:16+00:00
Next Scan 2024-07-14T08:01:16+00:00

Last Scan

Scanned2024-06-14T08:01:16+00:00
URL https://freedompet.com/robots.txt
Redirect https://www.freedompet.com/robots.txt
Redirect Domain www.freedompet.com
Redirect Base freedompet.com
Domain IPs 104.21.10.207, 172.67.190.254, 2606:4700:3030::6815:acf, 2606:4700:3032::ac43:befe
Redirect IPs 104.21.10.207, 172.67.190.254, 2606:4700:3030::6815:acf, 2606:4700:3032::ac43:befe
Response IP 172.67.190.254
Found Yes
Hash 88b4b99fb25fa24de85e5a1fcd25def998563c27789f945de72278fd389d0569
SimHash a6869d03c6f3

Groups

yandex
moget
ichiro
naverbot
yeti
baiduspider
baiduspider-video
baiduspider-image
sogou spider
youdaobot

Rule Path
Disallow /

*

Rule Path
Disallow /ajax
Disallow

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /*?*filter%5B

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • XM Symphony robots.txt file
  • The following spiders are considered aggressive and /or non-desirable.
  • The following rules allow all other user-agents, but with a crawl delay of 10 sec
  • NOTE by default this is set to NOT allow indexing of the site
  • when site goes live: Change "Disallow: /" to "Disallow:"
Back to top