Log In Sign Up

usf.edu
robots.txt

Robots Exclusion Standard data for usf.edu

Resource Scan

Scan Details

Site Domain usf.edu
Base Domain usf.edu
Scan Status Ok
Last Scan2024-10-21T03:57:42+00:00
Next Scan 2024-11-20T03:57:42+00:00

Last Scan

Scanned2024-10-21T03:57:42+00:00
URL https://usf.edu/robots.txt
Redirect https://www.usf.edu/robots.txt
Redirect Domain www.usf.edu
Redirect Base usf.edu
Domain IPs 131.247.1.40, 131.247.100.1, 52.141.216.229
Redirect IPs 52.149.184.58
Response IP 52.149.184.58
Found Yes
Hash c74367c53a1ea114e4a63faf6b5625583c8c58b534ef2dd5f2695af8aa01d6e1
SimHash 701019d1cff1

Groups

*

Rule Path
Disallow /images
Disallow /css
Disallow /aspnet_client
Disallow /javascripts
Disallow /snapshots
Disallow /web-templates
Disallow /inc
Disallow /utilities
Disallow /calendar
Disallow /it/archive
Disallow /test-bob
Disallow /student-affairs/tedx
Disallow /Atest.html
Disallow /jstest.asp
Disallow /About-USF/contact-send.asp
Disallow /About-USF/page-not-found.asp

ahrefsbot
ahrefssiteaudit

Rule Path
Allow /admissions

twitterbot

Rule Path
Allow /images
Back to top

Comments

  • robots.txt for USF.edu
  • list folders robots are not allowed to index
  • list specific files robots are not allowed to index
  • Ahrefs for Admissions
  • allow twitter to fetch images for news
  • End of robots.txt file
Back to top