Log In Sign Up

crefan.jp
robots.txt

Robots Exclusion Standard data for crefan.jp

Resource Scan

Scan Details

Site Domain crefan.jp
Base Domain crefan.jp
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-10-25T23:36:13+00:00
Next Scan 2025-01-23T23:36:13+00:00

Last Successful Scan

Scanned2024-06-28T23:34:39+00:00
URL https://crefan.jp/robots.txt
Domain IPs 118.27.18.204
Response IP 118.27.18.204
Found Yes
Hash e68a52cff2f96980193dd5a3e7fe0fe0ccacd596a998d3f8032c2e5267d3ed95
SimHash 305cd0c01f81

Groups

megalodon

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

bitlybot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

smrjbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

your-search-bot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

exabot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

webliobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

brandverity

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

amazonadbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 30

googlebot

Rule Path
Disallow /cads/compare/
Disallow /cads/blocks/
Back to top

Comments

  • User-agent: Twitterbot
  • Disallow: /
Back to top

Warnings

  • 2 invalid lines.
Back to top