twinavi.jp
robots.txt

Robots Exclusion Standard data for twinavi.jp

Resource Scan

Scan Details

Site Domain twinavi.jp
Base Domain twinavi.jp
Scan Status Ok
Last Scan2024-11-15T08:24:43+00:00
Next Scan 2024-11-22T08:24:43+00:00

Last Scan

Scanned2024-11-15T08:24:43+00:00
URL https://twinavi.jp/robots.txt
Domain IPs 23.203.75.254
Response IP 23.203.75.254
Found Yes
Hash 1b3faaf9bf27a0b149a23ccf9cb67d9719d2a932ecc6a9afbabd0347d867fb6b
SimHash 65669b417335

Groups

bingbot

Rule Path
Disallow /maintenance
Disallow /sorry

Other Records

Field Value
crawl-delay 20

msnbot

Rule Path
Disallow /maintenance
Disallow /sorry

mediapartners-google

Rule Path
Allow /
Disallow /maintenance
Disallow /sorry

googlebot-image

Rule Path
Disallow /maintenance
Disallow /sorry

googlebot

Rule Path
Disallow /maintenance
Disallow /sorry

googlebot-mobile

Rule Path
Disallow /maintenance
Disallow /sorry

twitterbot

Rule Path
Disallow /maintenance
Disallow /sorry

facebot

Rule Path
Disallow /maintenance
Disallow /sorry

slurp

Rule Path
Disallow /maintenance
Disallow /sorry

yahoo! slurp

Rule Path
Disallow /maintenance
Disallow /sorry

y!j-dlc/1.0

Rule Path
Disallow /maintenance
Disallow /sorry

y!j-bry/yatsh

Rule Path
Disallow /maintenance
Disallow /sorry

y!j-brj/yats

Rule Path
Disallow /maintenance
Disallow /sorry

popin_agent

Rule Path
Disallow /maintenance
Disallow /sorry

logly

Rule Path
Disallow /maintenance
Disallow /sorry

yahoocachesystem

Rule Path
Disallow /
Allow /topics/feed
Allow /topics/candidates
Allow /topics/reserved

pinterest

Rule Path
Disallow /matome
Disallow /search2
Disallow /twitter/search
Disallow /twitter/pickup
Disallow /twitter/weather
Disallow /twitter/conversation

steeler

Rule Path
Disallow /matome
Disallow /search2
Disallow /twitter/search
Disallow /twitter/pickup
Disallow /twitter/weather
Disallow /twitter/conversation

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /

Comments

  • Set for the general agent at the last.