Log In Sign Up

oakton.edu
robots.txt

Robots Exclusion Standard data for oakton.edu

Resource Scan

Scan Details

Site Domain oakton.edu
Base Domain oakton.edu
Scan Status Ok
Last Scan2024-09-24T00:10:41+00:00
Next Scan 2024-10-24T00:10:41+00:00

Last Scan

Scanned2024-09-24T00:10:41+00:00
URL https://oakton.edu/robots.txt
Redirect https://www.oakton.edu/robots.txt
Redirect Domain www.oakton.edu
Redirect Base oakton.edu
Domain IPs 192.234.16.8
Redirect IPs 192.234.16.8
Response IP 192.234.16.8
Found Yes
Hash b3f6808d0ace773788551cb31c4f35f4e9fcad49c74f6c4beafd9cbe923d7ff5
SimHash 2c051550d5d6

Groups

*
adsbot-google
googlebot

Rule Path
Disallow /my/
Disallow /resource/it/
Disallow /OnlineApp/
Disallow /user/

twitterbot

Rule Path
Disallow /my/
Disallow /resource/it/
Disallow /OnlineApp/
Disallow /user/
Back to top

Other Records

Field Value
sitemap https://www.oakton.edu/sitemap.xml
Back to top

Comments

  • This robots.txt file controls crawling of URLs
  • 10.20.2022
Back to top