travel.mdpr.jp
robots.txt

Robots Exclusion Standard data for travel.mdpr.jp

Resource Scan

Scan Details

Site Domain travel.mdpr.jp
Base Domain mdpr.jp
Scan Status Ok
Last Scan2024-10-23T00:28:57+00:00
Next Scan 2024-11-22T00:28:57+00:00

Last Scan

Scanned2024-10-23T00:28:57+00:00
URL https://travel.mdpr.jp/robots.txt
Domain IPs 13.114.110.228, 18.177.99.57, 35.76.188.173
Response IP 18.177.99.57
Found Yes
Hash bc2fa756e710e6ab1df8526fc57cf74158ef62dae098ba0e2f59c0d6ffb1e2a2
SimHash f234c7510f9b

Groups

*

Rule Path
Disallow /topics/check/
Disallow /url/link/
Disallow /url/outside/
Disallow /api/
Disallow /async/
Disallow /search
Disallow /spot/hotel_list
Disallow /spot/hotel_pickup
Disallow /spot/nearby_spot
Disallow /spot/related_topic

petalbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

eyemonit

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

Comments

  • https://webmaster.petalsearch.com/site/petalbot
  • https://bot.seekport.com
  • https://www.baidu.com/search/robots_jp.html
  • http://help.sogou.com/guide.html
  • http://help.sogou.com/guide.html
  • https://yandex.com/support/webmaster/controlling-robot/robots-txt.html
  • https://ahrefs.com/ja/robot
  • https://www.trendiction.com/bot
  • https://mj12bot.com/
  • https://www.mediatoolkit.com/robot
  • http://www.eyemon.it/
  • https://dataforseo.com/dataforseo-bot
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • https://ucri.nict.go.jp/icccrawler.html
  • http://femtosearch.com/