Log In Sign Up

stuportal.clcillinois.edu
robots.txt

Robots Exclusion Standard data for stuportal.clcillinois.edu

Resource Scan

Scan Details

Site Domain stuportal.clcillinois.edu
Base Domain clcillinois.edu
Scan Status Ok
Last Scan2025-08-30T12:12:56+00:00
Next Scan 2025-09-29T12:12:56+00:00

Last Scan

Scanned2025-08-30T12:12:56+00:00
URL https://stuportal.clcillinois.edu/robots.txt
Domain IPs 216.125.48.236
Response IP 216.125.48.236
Found Yes
Hash 69cec7be141cabd627b3642dd3f06dbb61ceea3cd8e9db5a309d1d44250d3ff7
SimHash 7819de1347b1

Groups

*

Rule Path
Disallow /

red

Rule Path
Disallow

screaming frog seo spider

Rule Path
Disallow

sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow

linkcheck by siteimprove.com

Rule Path
Disallow

image size by siteimprove.com

Rule Path
Disallow
Back to top

Comments

  • This file is used to block search engine indexing on all staging URLs
  • Allow www.redbot.org
  • Allow Screaming Frog
  • Allow Siteimprove
Back to top

Warnings

  • `noindex` is not a known field.
Back to top