Log In Sign Up

cropwise.com
robots.txt

Robots Exclusion Standard data for cropwise.com

Resource Scan

Scan Details

Site Domain cropwise.com
Base Domain cropwise.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-15T00:42:51+00:00
Next Scan 2026-03-15T00:42:51+00:00

Last Successful Scan

Scanned2022-11-06T21:50:46+00:00
URL https://www.cropwise.com/robots.txt
Response IP 104.18.18.38, 104.18.19.38
Found Yes
Hash 9b71d3fc1e23cf39a160029906d26264302738ea9c60653d368bb623b6901bf1
SimHash 3a949d08ce54

Groups

*

Rule Path
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F

Other Records

Field Value
crawl-delay 10

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /products
Disallow /products/
Disallow /search_api_autocomplete
Disallow /search_api_autocomplete/
Disallow /products/search
Disallow /products/search/
Disallow /search
Disallow /search/
Back to top

Other Records

Field Value
sitemap https://www.cropwise.com/sitemap.xml
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • For syntax checking, see:
  • http://www.frobee.com/robots-txt-check
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Added more lines from hook - syn_core_robotstxt.
  • blocking bad bots
Back to top