Log In Sign Up

animaldiversity.org
robots.txt

Robots Exclusion Standard data for animaldiversity.org

Resource Scan

Scan Details

Site Domain animaldiversity.org
Base Domain animaldiversity.org
Scan Status Ok
Last Scan2025-11-24T16:37:38+00:00
Next Scan 2025-12-24T16:37:38+00:00

Last Scan

Scanned2025-11-24T16:37:38+00:00
URL https://animaldiversity.org/robots.txt
Domain IPs 143.198.3.86
Response IP 143.198.3.86
Found Yes
Hash b42f95784809b77b87d2fa3b4b0ea0d6af4f9c2bc85f032e5ff7773bbfd553cc
SimHash 05a8c670213b

Groups

*

Rule Path
Disallow /mousetrap/
Disallow /skunkworks/
Disallow /mousetrap/courses/
Disallow /mousetrap2k2/
Disallow /workspaces/
Disallow /local/
Disallow /quaardvark/search/
Disallow /quaardvark/user/register/
Disallow /quaardvark/user/login/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 16
Back to top

Comments

  • Disallow: /site/accounts/classification/
  • Disallow: /resources/*.jpg
  • Disallow: /media/*/*.jpg
  • Allow: /media/*/*_th.jpg
  • Allow: */thumbnail.jpg
  • Allow: */badge.jpg
  • Allow: */small.jpg
Back to top