Log In Sign Up

biokids.umich.edu
robots.txt

Robots Exclusion Standard data for biokids.umich.edu

Resource Scan

Scan Details

Site Domain biokids.umich.edu
Base Domain umich.edu
Scan Status Ok
Last Scan2024-05-28T09:59:15+00:00
Next Scan 2024-06-04T09:59:15+00:00

Last Scan

Scanned2024-05-28T09:59:15+00:00
URL http://biokids.umich.edu/robots.txt
Redirect https://animaldiversity.org/robots.txt
Redirect Domain animaldiversity.org
Redirect Base animaldiversity.org
Domain IPs 141.211.4.201
Redirect IPs 174.138.63.167
Response IP 174.138.63.167
Found Yes
Hash b42f95784809b77b87d2fa3b4b0ea0d6af4f9c2bc85f032e5ff7773bbfd553cc
SimHash 05a8c670213b

Groups

*

Rule Path
Disallow /mousetrap/
Disallow /skunkworks/
Disallow /mousetrap/courses/
Disallow /mousetrap2k2/
Disallow /workspaces/
Disallow /local/
Disallow /quaardvark/search/
Disallow /quaardvark/user/register/
Disallow /quaardvark/user/login/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 16
Back to top

Comments

  • Disallow: /site/accounts/classification/
  • Disallow: /resources/*.jpg
  • Disallow: /media/*/*.jpg
  • Allow: /media/*/*_th.jpg
  • Allow: */thumbnail.jpg
  • Allow: */badge.jpg
  • Allow: */small.jpg
Back to top