Log In Sign Up

dietnavi.com
robots.txt

Robots Exclusion Standard data for dietnavi.com

Resource Scan

Scan Details

Site Domain dietnavi.com
Base Domain dietnavi.com
Scan Status Ok
Last Scan2024-09-25T18:09:16+00:00
Next Scan 2024-10-02T18:09:16+00:00

Last Scan

Scanned2024-09-25T18:09:16+00:00
URL https://dietnavi.com/robots.txt
Domain IPs 13.113.64.175, 52.69.208.30, 57.181.78.149
Response IP 13.113.64.175
Found Yes
Hash eb99fb629a781760a77cc1b667c4f4a6783ebe2293758118c16fd826998f0a21
SimHash 2004d0607583

Groups

bingbot

Rule Path
Allow /
Disallow /*

Other Records

Field Value
crawl-delay 3600000

baiduspider

Rule Path
Disallow /

linespider

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

linkfluence.com

Rule Path
Disallow /

yak

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

clipper

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

http://firstra.in/fr3_0

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

proximic

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

zoominfo

Rule Path
Disallow /

punkspider

Rule Path
Disallow /

punkspider.org

Rule Path
Disallow /

webdatastats

Rule Path
Disallow /

*

Rule Path
Allow /pc/
Allow /sp/
Allow /campaign/
Allow /rss/
Disallow /unsubscribe/
Back to top

Other Records

Field Value
sitemap https://dietnavi.com/sitemap.xml
Back to top

Comments

  • Bing
  • baidu
  • Naver / LINE
  • YaK/linkfluence.com
  • AdmantX
  • AhrefsBot
  • webmeup BLEX
  • Buck
  • Clipper
  • Common Crawl
  • dotbot
  • ExaBot
  • Femtosearch
  • firstrain
  • flamingosearch.com
  • grapeshot
  • NICT ICC
  • Magpie Brandwatch
  • Mappy Crawler
  • cyber.harvard.edu
  • MJ12bot
  • Nutch
  • comscore
  • Linguee
  • Semrush
  • sogou.com chinese search engine
  • toutia
  • Trendiction
  • trendkite
  • Yandex
  • ZoominfoBot, not related webmeeting system: zoom
  • Punkspider vulnerability scanner
  • WebDataStats, russian vulnerability scanner
Back to top