haitenjp.com
robots.txt

Robots Exclusion Standard data for haitenjp.com

Resource Scan

Scan Details

Site Domain haitenjp.com
Base Domain haitenjp.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-21T03:08:41+00:00
Next Scan 2024-11-19T03:08:41+00:00

Last Successful Scan

Scanned2024-03-31T23:19:01+00:00
URL https://haitenjp.com/robots.txt
Domain IPs 104.21.62.134, 172.67.136.45, 2606:4700:3033::ac43:882d, 2606:4700:3035::6815:3e86
Response IP 104.21.62.134
Found Yes
Hash 21aaa6049e906a330040e0d05bf916e3e0e1665b85b2cbb4ebaa7b6097c73183
SimHash 48098002ef9c

Groups

ahrefsbot
mj12bot
semrushbot
telegrambot
yandexbot
superfeedr
seznambot
linguee
slackbot
clark-crawler2
clark-crawler2
my-tiny-bot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login*
Allow *.css
Allow *.js