divusrawdogfood.com
robots.txt

Robots Exclusion Standard data for divusrawdogfood.com

Resource Scan

Scan Details

Site Domain divusrawdogfood.com
Base Domain divusrawdogfood.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-09T01:50:48+00:00
Next Scan 2024-11-07T01:50:48+00:00

Last Successful Scan

Scanned2024-03-20T00:15:42+00:00
URL https://divusrawdogfood.com/robots.txt
Redirect https://divusfoods.com/robots.txt
Redirect Domain divusfoods.com
Redirect Base divusfoods.com
Domain IPs 104.21.48.21, 172.67.176.6, 2606:4700:3035::6815:3015, 2606:4700:3035::ac43:b006
Redirect IPs 104.26.8.5, 104.26.9.5, 172.67.70.236, 2606:4700:20::681a:805, 2606:4700:20::681a:905, 2606:4700:20::ac43:46ec
Response IP 104.26.9.5
Found Yes
Hash 3c33054d77ca03e633e8842af1860054c31857e301117c7312f4de6d830d8e4f
SimHash 6f30dce609a0

Groups

*

Rule Path
Disallow /tags
Disallow /tags/*
Disallow /contacts/*
Disallow /checkout
Disallow /checkout/*
Disallow /cart
Disallow /cart/*
Disallow /wishlist/*
Disallow /bbt
Disallow /bbt/*
Disallow /checkout
Disallow /checkout/*
Disallow /cart
Disallow /cart/*
Disallow /wishlist/*
Disallow /bbt
Disallow /bbt/*

Other Records

Field Value
crawl-delay 3
crawl-delay 3