megashkola.com
robots.txt

Robots Exclusion Standard data for megashkola.com

Resource Scan

Scan Details

Site Domain megashkola.com
Base Domain megashkola.com
Scan Status Ok
Last Scan2024-11-04T23:39:15+00:00
Next Scan 2024-11-11T23:39:15+00:00

Last Scan

Scanned2024-11-04T23:39:15+00:00
URL https://megashkola.com/robots.txt
Domain IPs 104.21.85.213, 172.67.211.104, 2606:4700:3035::6815:55d5, 2606:4700:3035::ac43:d368
Response IP 172.67.211.104
Found Yes
Hash 72f3fe115a93b300c27b1f42d320f8da7354036d970d1e79d3c9938f942bf53f
SimHash 6830b6d2ecd3

Groups

*

Rule Path
Allow /ads.txt
Disallow /advertisements/gift_clicks
Disallow /cdn-cgi/l/email-protection
Disallow /login?*
Disallow /signup?*
Disallow /question/add?*
Disallow *?*
Disallow /files/*
Disallow /profile/*

yandex

Rule Path
Disallow /advertisements/gift_clicks
Disallow /app/ask?*
Disallow /buddies/invite/
Disallow /buddies_new/invite/
Disallow /cdn-cgi/l/email-protection
Disallow /login?*
Disallow /question/add?*
Disallow /signup?*
Disallow /tasks/prev_task/
Disallow /tasks/next_task/
Disallow /tasks/latex/
Disallow /tasks/solve_dynamic/
Disallow /users/thank/
Disallow /users/view_awards/

semrushbot-sa

Rule Path Comment
Disallow / Semrush
Allow /ads.txt -

semrushbot

Rule Path Comment
Disallow / Semrush
Allow /ads.txt -

rogerbot

Rule Path Comment
Disallow / MOZ
Allow /ads.txt -

dotbot

Rule Path Comment
Disallow / MOZ
Allow /ads.txt -

blexbot

Rule Path Comment
Disallow / Webmeup.com
Allow /ads.txt -

spbot

Rule Path Comment
Disallow / Openlinkprofiler
Allow /ads.txt -

seodiver

Rule Path Comment
Disallow / SEOdiver
Allow /ads.txt -

dataprovider

Rule Path Comment
Disallow / DataProvider.com
Allow /ads.txt -

magpie-crawler

Rule Path Comment
Disallow / BrandWatch.com
Allow /ads.txt -

getintent crawler

Rule Path
Disallow /
Allow /ads.txt

grapeshot

Rule Path
Disallow

doubleverify

Rule Path
Disallow

white ops

Rule Path
Disallow

moatbot

Rule Path
Disallow

ias_crawler

Rule Path
Disallow

forensiq

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

leikibot

Rule Path
Disallow

baidu-yunguance-scanbot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-slabot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-perfbot(ce.baidu.com)

Rule Path
Disallow

baidu-yunguance-vsbot(ce.baidu.com)

Rule Path
Disallow

seznambot

Rule Path
Disallow /
Allow /ads.txt

sogou web spider

Rule Path
Disallow /
Allow /ads.txt

baiduspider

Rule Path
Disallow /
Allow /ads.txt

naverbot

Rule Path
Disallow /
Allow /ads.txt

yeti

Rule Path
Disallow /
Allow /ads.txt

coccocbot-web

Rule Path
Disallow /
Allow /ads.txt

qwantify

Rule Path
Disallow /
Allow /ads.txt

exabot

Rule Path
Disallow /
Allow /ads.txt

linguee

Rule Path Comment
Disallow / Language tool
Allow /ads.txt -

surdotlybot

Rule Path Comment
Disallow / Sur.ly
Allow /ads.txt -

bubing

Rule Path Comment
Disallow / Bubing academic crawler
Allow /ads.txt -

twitterbot

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

facebot

Rule Path
Disallow

Other Records

Field Value
sitemap https://megashkola.com/sitemap.xml

Comments

  • Disallow Marketing bots
  • Disallow exotic search engine crawlers
  • Disallow other crawlers
  • Good bots whitelisting:

Warnings

  • 3 invalid lines.
  • `clean-param` is not a known field.
  • `host` is not a known field.