linuxreviews.org
robots.txt

Robots Exclusion Standard data for linuxreviews.org

Resource Scan

Scan Details

Site Domain linuxreviews.org
Base Domain linuxreviews.org
Scan Status Ok
Last Scan2025-06-02T11:44:52+00:00
Next Scan 2025-06-09T11:44:52+00:00

Last Scan

Scanned2025-06-02T11:44:52+00:00
URL https://linuxreviews.org/robots.txt
Domain IPs 2001:470:1f0b:801::10, 85.25.199.78
Response IP 85.25.199.78
Found Yes
Hash bf19566930483cc3a6212f0c681f9b10d27ebba4dae195bfcd2e7cfd4d5ff8db
SimHash 220bd5745f91

Groups

mediapartners-google*

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Allow /api.php?
Allow /api.php?action=
Allow /api.php?*&action=
Allow /w/load.php?
Allow /w/images/
Allow /w/extensions/TimedMediaHandler/resources/mwembed/
Allow /w/extensions/Comments/resources/images/
Allow /w/skins/WPtouch/resources/images/
Allow /w/index.php?title=Kategori%3A
Allow /w/index.php?title=Category%3A
Disallow /w/
Disallow /api/
Disallow /trap/
Disallow /Special%3A
Disallow /Spesial%3A

Other Records

Field Value
crawl-delay 3

Other Records

Field Value
sitemap https://linuxreviews.org/sitemap.xml

Comments

  • MediaWiki robots.txt v2.0.0
  • See https://linuxreviews.org/Web_crawlers