urbangyal.com
robots.txt

Robots Exclusion Standard data for urbangyal.com

Resource Scan

Scan Details

Site Domain urbangyal.com
Base Domain urbangyal.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-28T12:30:02+00:00
Next Scan 2024-10-26T12:30:02+00:00

Last Successful Scan

Scanned2023-04-06T05:13:33+00:00
URL https://urbangyal.com/robots.txt
Redirect http://urbangyal.com/robots.txt
Domain IPs 104.21.1.168, 172.67.129.165, 2606:4700:3033::6815:1a8, 2606:4700:3033::ac43:81a5
Response IP 104.21.1.168
Found Yes
Hash 5fe0c524da895338d85a1baebb3dd07e71e56abe0ae86283b3133e48b1b78537
SimHash 6200dc115330

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Comments

  • slow down Yahoo