pedigreeonline.com
robots.txt

Robots Exclusion Standard data for pedigreeonline.com

Resource Scan

Scan Details

Site Domain pedigreeonline.com
Base Domain pedigreeonline.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-10T04:33:09+00:00
Next Scan 2026-01-08T04:33:09+00:00

Last Successful Scan

Scanned2025-03-15T03:53:22+00:00
URL https://pedigreeonline.com/robots.txt
Redirect https://www.pedigreeonline.com/robots.txt
Redirect Domain www.pedigreeonline.com
Redirect Base pedigreeonline.com
Domain IPs 104.26.12.2, 104.26.13.2, 172.67.72.110, 2606:4700:20::681a:c02, 2606:4700:20::681a:d02, 2606:4700:20::ac43:486e
Redirect IPs 104.26.12.2, 104.26.13.2, 172.67.72.110, 2606:4700:20::681a:c02, 2606:4700:20::681a:d02, 2606:4700:20::ac43:486e
Response IP 104.26.13.2
Found Yes
Hash 622f04931f1892549ad930b848c5498107495bb0565eb23967b3490de172471d
SimHash 4b5ed472c690

Groups

*

Rule Path
Allow /

ahrefssiteaudit
ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 4

semrushbot
mj12bot
dotbot
dataforseobot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

baiduspider
yandex
sogou
blexbot
petalbot
velenpublicwebcrawler
dataforseobot
netcraftsurveyagent
barkrowler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.pedigreeonline.com/sitemaps/www/sitemap.xml

Warnings

  • 1 invalid line.