gman.jp
robots.txt

Robots Exclusion Standard data for gman.jp

Resource Scan

Scan Details

Site Domain gman.jp
Base Domain gman.jp
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-10-01T22:50:29+00:00
Next Scan 2024-10-31T22:50:29+00:00

Last Successful Scan

Scanned2024-03-04T19:53:56+00:00
URL https://gman.jp/robots.txt
Domain IPs 153.122.13.66
Response IP 153.122.13.66
Found Yes
Hash 66efd9cfb368970e1b3483d2e2e89a8a8d22e5a94c03619a4ac8d04d41781b5b
SimHash 76175850c299

Groups

*

Rule Path
Disallow /trcm
Disallow /post
Disallow /regist
Disallow /user
Disallow /remark
Disallow /comment.php
Disallow /link.php
Disallow /shio2
Disallow /ship

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

babbar

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

steeler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

newspaper

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://gman.jp/google_sitemap.xml

Comments

  • Disallow: /login
  • 2021-04-11 test コメントアウト
  • User-agent: msnbot
  • Disallow: /
  • User-agent: bingbot
  • Disallow: /
  • 2021-07-02 add.
  • 最大が 30 らしい