en.linuxreviews.org
robots.txt

Robots Exclusion Standard data for en.linuxreviews.org

Resource Scan

Scan Details

Site Domain en.linuxreviews.org
Base Domain linuxreviews.org
Scan Status Ok
Last Scan2025-06-07T12:28:51+00:00
Next Scan 2025-06-14T12:28:51+00:00

Last Scan

Scanned2025-06-07T12:28:51+00:00
URL http://en.linuxreviews.org/robots.txt
Redirect https://linuxreviews.org/robots.txt
Redirect Domain linuxreviews.org
Redirect Base linuxreviews.org
Domain IPs 2001:470:1f0b:801::10, 85.25.199.78
Redirect IPs 2001:470:1f0b:801::10, 85.25.199.78
Response IP 85.25.199.78
Found Yes
Hash bf19566930483cc3a6212f0c681f9b10d27ebba4dae195bfcd2e7cfd4d5ff8db
SimHash 220bd5745f91

Groups

mediapartners-google*

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Allow /api.php?
Allow /api.php?action=
Allow /api.php?*&action=
Allow /w/load.php?
Allow /w/images/
Allow /w/extensions/TimedMediaHandler/resources/mwembed/
Allow /w/extensions/Comments/resources/images/
Allow /w/skins/WPtouch/resources/images/
Allow /w/index.php?title=Kategori%3A
Allow /w/index.php?title=Category%3A
Disallow /w/
Disallow /api/
Disallow /trap/
Disallow /Special%3A
Disallow /Spesial%3A

Other Records

Field Value
crawl-delay 3

Other Records

Field Value
sitemap https://linuxreviews.org/sitemap.xml

Comments

  • MediaWiki robots.txt v2.0.0
  • See https://linuxreviews.org/Web_crawlers