Log In Sign Up

eckerd.edu
robots.txt

Robots Exclusion Standard data for eckerd.edu

Resource Scan

Scan Details

Site Domain eckerd.edu
Base Domain eckerd.edu
Scan Status Ok
Last Scan2024-04-22T08:09:51+00:00
Next Scan 2024-05-22T08:09:51+00:00

Last Scan

Scanned2024-04-22T08:09:51+00:00
URL https://eckerd.edu/robots.txt
Redirect https://www.eckerd.edu/robots.txt
Redirect Domain www.eckerd.edu
Redirect Base eckerd.edu
Domain IPs 13.33.88.10, 13.33.88.102, 13.33.88.66, 13.33.88.80
Redirect IPs 13.33.88.10, 13.33.88.102, 13.33.88.66, 13.33.88.80
Response IP 13.33.88.102
Found Yes
Hash 64faadaf1357503d592cc083cda57bcfe2d7a76d161358a37a19cc7e07136285
SimHash 6a14d7306bf5

Groups

*

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin
Disallow /wp-login.php
Disallow /?s=
Disallow /search
Disallow /bot/
Disallow /career-services/wcidwtm/
Disallow /updatemyinfo
Disallow /updatemyinfo/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.eckerd.edu/sitemap_index.xml
Back to top

Comments

  • This tells all web crawlers the pages to overtly look for
  • This section indicates that crawlers need to have a 10 second delay between pages
  • All site directories are followable
  • Restricted access to search results and login pages
Back to top