Log In Sign Up

meganesuper.co.jp
robots.txt

Robots Exclusion Standard data for meganesuper.co.jp

Resource Scan

Scan Details

Site Domain meganesuper.co.jp
Base Domain meganesuper.co.jp
Scan Status Ok
Last Scan2024-09-09T00:57:33+00:00
Next Scan 2024-10-09T00:57:33+00:00

Last Scan

Scanned2024-09-09T00:57:33+00:00
URL https://www.meganesuper.co.jp/robots.txt
Domain IPs 18.178.224.142, 52.198.44.7, 54.248.93.216
Response IP 18.178.224.142
Found Yes
Hash 295a7f345fc5222be0b3099a2527a51181090b829e4a72cd39e306d81fb2e437
SimHash 03077163e3e2

Groups

exb language crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

nutch

Rule Path
Disallow /

feedbooster

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

bpimagewalker/2.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

mixbot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

heritrix

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

yandexbot

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

steeler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

linkdexbot/

Rule Path
Disallow /

cms crawler

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

kaloogabot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

exabot

Rule Path
Disallow /

exabot-images

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

yeti

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Back to top

Other Records

Field Value
sitemap https://www.meganesuper.co.jp/sitemap.xml
Back to top

Comments

  • language
  • seo
  • seo
  • seo moz
  • news portal
  • Apache Nutch based
  • news
  • search engine www.sengine.info
  • plagiarism check
  • seo
  • search engine
  • seo
  • seo
  • brand protection
  • seo
  • search engine
  • "analyses the structure of the WWW"
  • Web crawler
  • part of Ware Bay Best Buys Search engine
  • kostenlose Backlinkchecker von Torsten Rückert Internetdiestleistungen
  • the Internet Archive's open-source, extensible, scalable, archival-quality Web crawler
  • search engine
  • ??? search engine
  • scalable, fully distributed crawler
  • chinese search engine
  • chinese search engine
  • Mixdata : data for big business
  • search engine
  • search engine
  • tiscali.it search bot
  • panscient.com
  • SEO services
  • The web search API
Back to top