Log In Sign Up

petfinder.my
robots.txt

Robots Exclusion Standard data for petfinder.my

Resource Scan

Scan Details

Site Domain petfinder.my
Base Domain petfinder.my
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-23T20:50:46+00:00
Next Scan 2024-10-21T20:50:46+00:00

Last Successful Scan

Scanned2023-12-27T16:55:25+00:00
URL https://petfinder.my/robots.txt
Redirect https://www.petfinder.my/robots.txt
Redirect Domain www.petfinder.my
Redirect Base petfinder.my
Domain IPs 104.21.12.40, 172.67.151.134, 2606:4700:3032::6815:c28, 2606:4700:3035::ac43:9786
Redirect IPs 104.21.12.40, 172.67.151.134, 2606:4700:3032::6815:c28, 2606:4700:3035::ac43:9786
Response IP 172.67.151.134
Found Yes
Hash 32afb8c675eb17d9c32c5cff993cddf46e6ce270006c94524ce662a3657da3cb
SimHash 2b4ad280c1a1

Groups

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

facebookexternalhit/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

facebookexternalhit/*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yahoo!-adcrawler

Rule Path
Disallow /

cloudflare

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • Disallow: /wagazine/
Back to top