Log In Sign Up

bluesnews.com
robots.txt

Robots Exclusion Standard data for bluesnews.com

Resource Scan

Scan Details

Site Domain bluesnews.com
Base Domain bluesnews.com
Scan Status Ok
Last Scan2024-09-25T19:01:21+00:00
Next Scan 2024-10-02T19:01:21+00:00

Last Scan

Scanned2024-09-25T19:01:21+00:00
URL https://bluesnews.com/robots.txt
Redirect https://www.bluesnews.com/robots.txt
Redirect Domain www.bluesnews.com
Redirect Base bluesnews.com
Domain IPs 23.226.128.58
Redirect IPs 23.226.128.58
Response IP 23.226.128.58
Found Yes
Hash de457271a35539a31491f71ee491fe9da7dcf645888c7e823969dbbab8c6678a
SimHash 510c7dc3edf0

Groups

*

Rule Path
Disallow /files/descent/
Disallow /cgi-bin/board.pl?action=postmessage
Disallow /cgi-bin/board.pl?action=quotemessage
Disallow /cgi-bin/board.pl?action=editpost
Disallow /cgi-bin/board.pl?action=deletepost
Disallow /cgi-bin/board.pl?action=reportpost
Disallow /cgi-bin/board.pl?action=searchconfirm
Disallow /cgi-bin/board.pl?action=markread
Disallow /cgi-bin/board.pl?action=ignore
Disallow /cgi-bin/board.pl?action=unignore
Disallow /cgi-bin/board.pl?action=viewignores
Disallow /cgi-bin/edit.pl
Disallow /cgi-bin/delete.pl
Disallow /cgi-bin/deletethread.pl
Disallow /cgi-bin/deletescreens.pl
Disallow /cgi-bin/user.pl
Disallow /cgi-bin/myblues.pl
Disallow /cgi-bin/lanparties.pl?admin=true
Disallow /cgi-bin/lanparties.pl?addparty=true
Disallow /cgi-bin/blammo.pl?mode=archive&action=runsearch
Disallow /cgi-bin/blammo.pl?mode=mboard
Disallow /cgi-bin/planboard.pl
Disallow /cgi-bin/shred.pl
Disallow /cgi-bin/board.pl%0A
Disallow /cgi-bin/boa%5Cx72d.pl
Disallow /cgi-bin/texts.pl

Other Records

Field Value
crawl-delay 5

megaindex

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

jorgee

Rule Path
Disallow /

seekport

Rule Path
Disallow /

bluechipbacklinks

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

test-bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

aiohttp/

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.bluesnews.com/sitemap.xml
Back to top

Comments

  • robots.txt exclusion file for www.bluesnews.com
  • 30-Jan-2002 fpv Created.
  • Announce Sitemap
  • Do not remove! Log in and look in this directory for details! -Furn
  • Disallow: /files/
  • Block bots from input forms
  • Block bots from obsolete paths
  • Block Googlebot from bogus scripts
  • Block Yahoo Slurp from bogus script
  • Block Alexa (ia_archiver) from bogus script
  • Block Yandex from bogus script
  • Block MegaIndex for slurping too much too fast
  • Block bogus slurper (heritrix/1.14.1 +http://www.iseclab.org)
  • Block broken slurper (iCcrawler - iCjobs Stellenangebote Jobs; http://www.icjobs.de)
  • Block bogus slurper
  • Block bogus slurper
  • Block bogus slurper
  • Block bogus slurper
  • Block bogus slurper
  • https://linuxreviews.org/Web_crawlers:
Back to top