Log In Sign Up

spaceweatherlive.com
robots.txt

Robots Exclusion Standard data for spaceweatherlive.com

Resource Scan

Scan Details

Site Domain spaceweatherlive.com
Base Domain spaceweatherlive.com
Scan Status Ok
Last Scan2024-10-05T00:07:12+00:00
Next Scan 2024-10-12T00:07:12+00:00

Last Scan

Scanned2024-10-05T00:07:12+00:00
URL https://spaceweatherlive.com/robots.txt
Domain IPs 104.18.30.18, 104.18.31.18, 2606:4700::6812:1e12, 2606:4700::6812:1f12
Response IP 104.18.31.18
Found Yes
Hash c495f95c140b95d269b9529963ba66fc238601e3a151d18610c71fc6531f3023
SimHash c8349cf39188

Groups

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

ranksonicbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

*

Rule Path
Disallow /community/startTopic/
Disallow /community/*?do=add
Disallow /community/*?do=submit
Disallow /community/discover/unread/
Disallow /community/markallread/
Disallow /community/staff/
Disallow /community/online/
Disallow /community/discover/
Disallow /community/leaderboard/
Disallow /community/search/
Disallow /community/*?advancedSearchForm=
Disallow /community/register/
Disallow /community/lostpassword/
Disallow /community/login/
Disallow /community/*?sortby=
Disallow /community/*?filter=
Disallow /community/*?tab=comments
Disallow /community/*?do=email
Disallow /community/*?do=findComment
Disallow /community/*?do=getLastComment
Disallow /community/*?do=getNewComment
Disallow /community/profile/
Back to top

Other Records

Field Value
sitemap https://www.spaceweatherlive.com/sitemap.php
Back to top

Comments

  • http://www.opensiteexplorer.org/dotbot
  • http://openlinkprofiler.org/bot
  • http://www.crazywebcrawler.com/
  • https://ranksonic.com/ranksonic_bot.html
  • http://www.seokicks.de/robot.html
  • These fools don't even respect robots.txt
  • http://www.picsearch.com/bot.html
  • domainreanimator.com
  • Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
  • http://megaindex.com/crawler
  • Rules for Invision Community (https://invisioncommunity.com)
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
  • Sitemap URL
Back to top