Log In Sign Up

scienceaid.net
robots.txt

Robots Exclusion Standard data for scienceaid.net

Resource Scan

Scan Details

Site Domain scienceaid.net
Base Domain scienceaid.net
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-08-29T05:50:07+00:00
Next Scan 2024-11-27T05:50:07+00:00

Last Successful Scan

Scanned2024-05-02T05:48:34+00:00
URL https://scienceaid.net/robots.txt
Domain IPs 178.32.62.241
Response IP 178.32.62.241
Found Yes
Hash 66c8e7a1e14b1fb98d0cf6da649b5abf260893ff38494ddadd53640a5ed17dd8
SimHash ac10098947f7

Groups

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

hmse_robot

Rule Path
Disallow /

googlebot

Rule Path
Allow /Special%3ACategorylisting
Allow /Special%3AAllpages
Allow /Special%3APopularpages
Allow /Special%3ANewpages
Allow /Special%3ASitemap
Allow /Special%3ALSearch
Allow /Special%3AUserlogin
Allow /Special%3AEmailuser
Allow /Special%3AThankAuthors
Allow /Special%3ARCTiles
Allow /index.php?*action=credits
Allow /index.php?*printable
Allow /index.php?action=ajax&rs=wfShowMore
Disallow /w/
Disallow /forum/
Disallow /index.php
Disallow /*feed%3Drss
Disallow /*action%3Ddelete
Disallow /*action%3Dedit
Disallow /*action%3Dhistory
Disallow /*action%3Dprotect
Disallow /*action%3Dwatch
Disallow /Special%3A
Disallow /api.php
Disallow /images/samplepdfs
Disallow /images/sampledocs*pdf
Disallow /images/sampledocs*doc
Disallow /images/sampledocs*txt

googlebot-mobile

Rule Path
Allow /Special%3ACategorylisting
Allow /Special%3AAllpages
Allow /Special%3APopularpages
Allow /Special%3ANewpages
Allow /Special%3ASitemap
Allow /Special%3ALSearch
Allow /Special%3AUserlogin
Allow /Special%3AIpblocklist
Allow /Special%3ABlockip
Allow /Special%3AEmailuser
Allow /Special%3AThankAuthors
Allow /Special%3ARCTiles
Allow /index.php?*action=credits
Allow /index.php?*printable
Allow /index.php?title=Special%3ABooksources
Disallow /w/
Disallow /forum/
Disallow /index.php
Disallow /*feed%3Drss
Disallow /*action%3Ddelete
Disallow /*action%3Dedit
Disallow /*action%3Dhistory
Disallow /*action%3Dprotect
Disallow /*action%3Dwatch
Disallow /Special%3A
Disallow /api.php
Disallow /images/samplepdfs
Disallow /images/sampledocs*pdf
Disallow /images/sampledocs*doc
Disallow /images/sampledocs*txt

*

Rule Path
Allow /Special%3ACategorylisting
Allow /Special%3AAllpages
Allow /Special%3APopularpages
Allow /Special%3ANewpages
Allow /Special%3ASitemap
Allow /Special%3ALSearch
Disallow /w/
Disallow /forum/
Disallow /index.php
Disallow /*feed%3Drss
Disallow /*action%3Ddelete
Disallow /*action%3Dedit
Disallow /*action%3Dhistory
Disallow /*action%3Dprotect
Disallow /*action%3Dwatch
Disallow /Special%3A
Disallow /images/samplepdfs
Disallow /images/sampledocs*pdf
Disallow /images/sampledocs*doc
Disallow /images/sampledocs*txt
Back to top

Other Records

Field Value
sitemap http://scienceaid.net/sitemap_index.xml
Back to top

Comments

  • robots.txt should never be edited directly.
  • The .htaccess file determines whether it is a production or development environment and redirects appropriately to the correct dev vs prod file.
  • robotsproduction.txt should also never be edited directly.
  • robotsproduction.template should be modified instead, so that the scienceaid.net value below can be filled automatically by index.php
  • IMPORTANT!!! READ THE ABOVE LINES TO MAKE SURE YOU ARE MODIFYING THE CORRECT FILE!!!
  • robots.txt for http://scienceaid.net
  • based on wikipedia.org's robots.txt
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • If your bot supports such a thing using the 'Crawl-delay' or another
  • instruction, please let us know. We can add it to our robots.txt.
  • Friendly, low-speed bots are welcome viewing article pages, but not
  • dynamically-generated pages please. Article pages contain our site's
  • real content.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • wget in recursive mode uses too many resources for us.
  • Please read the man page and use it properly; there is a
  • --wait option you can use to set the delay between hits,
  • for instance. Please wait 3 seconds between each request.
  • Doesn't follow robots.txt anyway, but...
  • Requests many pages per second
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • these rules should be identical to the User-agent: Googlebot rules
Back to top

Warnings

  • `noindex` is not a known field.
Back to top