Log In Sign Up

darkmatters.org
robots.txt

Robots Exclusion Standard data for darkmatters.org

Resource Scan

Scan Details

Site Domain darkmatters.org
Base Domain darkmatters.org
Scan Status Ok
Last Scan2025-06-20T10:06:14+00:00
Next Scan 2025-06-27T10:06:14+00:00

Last Scan

Scanned2025-06-20T10:06:14+00:00
URL https://darkmatters.org/robots.txt
Domain IPs 104.21.7.127, 172.67.155.120, 2606:4700:3033::6815:77f, 2606:4700:3034::ac43:9b78
Response IP 104.21.7.127
Found Yes
Hash 51604de045553fedcdeac90d2f68af011785e94bdf806747eb4fb1cfba0f30bb
SimHash b6100349c4f5

Groups

*

Rule Path
Disallow /forums/admin/
Disallow /forums/cache/
Disallow /forums/converge_local/
Disallow /forums/hooks/
Disallow /forums/ips_kernel/
Disallow /forums/retail/
Disallow /forums/public/style_captcha/
Disallow /forums/index.php?app=core&module=task
Disallow /forums/index.php?app=forums&module=moderate&section=moderate
Disallow /forums/index.php?app=forums&module=extras&section=forward
Disallow /forums/index.php?app=members&module=chat
Disallow /forums/index.php?&unlockUserAgent=1
Disallow /forums/*%26module%3Dsearch
Disallow /forums/*%26search_tags%3D
Disallow /forums/*app%3Dcore%26module%3Dglobal%26section%3Dreputation
Disallow /forums/*app%3Dcore%26module%3Dusercp
Disallow /forums/*app%3Dmembers%26module%3Dmessaging
Disallow /forums/*%26p%3D
Disallow /forums/*%26pid%3D
Disallow /forums/*%26hl%3D
Disallow /forums/*%26start%3D
Disallow /forums/*view__getnewpost$
Disallow /forums/*view__getlastpost$
Disallow /forums/*view__old$
Disallow /forums/*view__new$
Disallow /forums/*view__getfirst$
Disallow /forums/*view__getprevious$
Disallow /forums/*view__getnext$
Disallow /forums/*view__getlast$
Disallow /forums/*%26view%3Dgetnewpost$
Disallow /forums/*%26view%3Dgetlastpost$
Disallow /forums/*%26view%3Dold$
Disallow /forums/*%26view%3Dnew$
Disallow /forums/*%26view%3Dgetfirst$
Disallow /forums/*%26view%3Dgetprevious$
Disallow /forums/*%26view%3Dgetnext$
Disallow /forums/*%26view%3Dgetlast$
Disallow /forums/*?s=
Disallow /forums/*%26s%3D
Disallow /forums/index.php?app=core&module=global&section=login&do=deleteCookies
Disallow /forums/index.php?app=forums&module=extras&section=rating
Disallow /forums/index.php?app=forums&module=forums&section=markasread
Disallow /forums/*%26do%3Dwho
Disallow /forums/*%26section%3Ddname

scrapy

Rule Path
Disallow /

scrapy/2.11.2

Rule Path
Disallow /

googleother

Rule Path
Disallow /

*

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot/7.0

Rule Path
Disallow /

bingbot/2.0

Rule Path
Disallow /

googlebot/2.1

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow

fasterfox

Rule Path
Disallow /

israbot

Rule Path
Disallow

orthogaffe

Rule Path
Disallow

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow /forums/
Disallow /testforum/
Disallow /t/
Disallow /test/
Disallow /SacredWiki/Especial%3ASearch
Disallow /SacredWiki/Especial%3ASearch
Disallow /SacredWiki/Special%3ARandom
Disallow /SacredWiki/Special%3ARandom
Disallow /SacredWiki/Special%3ASearch
Disallow /SacredWiki/Special%3ASearch
Disallow /SacredWiki/Spesial%3ASearch
Disallow /SacredWiki/Spesial%3ASearch
Disallow /SacredWiki/Spezial%3ASearch
Disallow /SacredWiki/Spezial%3ASearch
Disallow /SacredWiki/Specjalna%3ASearch
Disallow /SacredWiki/Specjalna%3ASearch
Disallow /SacredWiki/Speciaal%3ASearch
Disallow /SacredWiki/Speciaal%3ASearch
Disallow /SacredWiki/Speciaal%3ARandom
Disallow /SacredWiki/Speciaal%3ARandom
Disallow /SacredWiki/Speciel%3ASearch
Disallow /SacredWiki/Speciel%3ASearch
Disallow /SacredWiki/Speciale%3ASearch
Disallow /SacredWiki/Speciale%3ASearch
Disallow /SacredWiki/Istimewa%3ASearch
Disallow /SacredWiki/Istimewa%3ASearch
Disallow /SacredWiki/Toiminnot%3ASearch
Disallow /SacredWiki/Toiminnot%3ASearch
Disallow /SacredWiki/%D8%AE%D8%A7%D8%B5%3ASearch
Disallow /SacredWiki/%D8%AE%D8%A7%D8%B5%3ASearch

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Comments

  • Wikipedia work bots:
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Sorry, wget in its recursive mode is a frequent problem.
  • Please read the man page and use it properly; there is a
  • --wait option you can use to set the delay between hits,
  • for instance.
  • The 'grub' distributed client has been *very* poorly behaved.
  • Doesn't follow robots.txt anyway, but...
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • Don't allow the wayback-maschine to index user-pages
  • Disallow: /SacredWiki/User
  • Disallow: /SacredWiki/Benutzer
  • Friendly, low-speed bots are welcome viewing article pages, but not
  • dynamically-generated pages please.
  • Inktomi's "Slurp" can read a minimum delay between hits; if your
  • bot supports such a thing using the 'Crawl-delay' or another
  • instruction, please let us know.
  • ar:
Back to top