city.living.jp
robots.txt

Robots Exclusion Standard data for city.living.jp

Resource Scan

Scan Details

Site Domain city.living.jp
Base Domain living.jp
Scan Status Ok
Last Scan2024-06-13T13:46:52+00:00
Next Scan 2024-07-13T13:46:52+00:00

Last Scan

Scanned2024-06-13T13:46:52+00:00
URL https://city.living.jp/robots.txt
Domain IPs 133.130.60.96, 202.238.245.108
Response IP 202.238.245.108
Found Yes
Hash b236f1369afc724abb8405a906a87ccf439fb704827d26914561455b54cc9f2d
SimHash 42ac4270e3a0

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /styleguide
Disallow */feed$
Allow /wp-admin/admin-ajax.php
Allow /wp-includes/js/
Allow /wp-includes/images/
Allow /wp-includes/css/
Allow /wp-includes/fonts/

slurp

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

mappy

Rule Path
Disallow /

yamanalab-robot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

onefuncbot

Rule Path
Disallow /

yak

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

toutiaospider/1.0

Rule Path
Disallow /

yak/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

trendictionbot0.5.0

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

applebot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

*

Rule Path
Allow /

Comments

  • http://www.semrush.com/bot.html
  • http://www.grapeshot.com/crawler/
  • https://www.comscore.com/Web-Crawler
  • http://mappydata.net/bot/
  • http://mj12bot.com/
  • http://www.baidu.com/search/spider.html
  • http://yandex.com/bots
  • http://webmeup-crawler.com/
  • http://www.bombora.com
  • http://www.eyeota.com
  • http://mappydata.net
  • Other ANY OK