uwlax.edu
robots.txt

Robots Exclusion Standard data for uwlax.edu

Resource Scan

Scan Details

Site Domain uwlax.edu
Base Domain uwlax.edu
Scan Status Ok
Last Scan2024-10-26T13:14:06+00:00
Next Scan 2024-11-25T13:14:06+00:00

Last Scan

Scanned2024-10-26T13:14:06+00:00
URL https://uwlax.edu/robots.txt
Redirect https://www.uwlax.edu/robots.txt
Redirect Domain www.uwlax.edu
Redirect Base uwlax.edu
Domain IPs 138.49.101.136
Redirect IPs 138.49.101.136
Response IP 138.49.101.136
Found Yes
Hash 2a14e7dbfd1f72aae56cdc77ea6253d65bcacacdd64af8cbf122d499ec0e7745
SimHash 92141d02c6b0

Groups

teoma

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

scirus

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Disallow /workarea/
Disallow /widgets/
Disallow /logs/
Disallow /masterpages/
Disallow /app_browsers/
Disallow /app_code/
Disallow /app_data/
Disallow /app_globalresources/
Disallow /app_webreferences
Disallow /assets/
Disallow /assetmanagement/
Disallow /xmlfiles/
Disallow /sandbox1/
Disallow /sandbox/
Disallow /Sandbox/
Disallow /archive/
Disallow /Archive/
Disallow /uploadedfiles/
Disallow /uploadedFiles/
Disallow /UploadedFiles/
Disallow /uploadedimages/
Disallow /uploadedImages/
Disallow /UploadedImages/
Disallow /recsports/app/
Disallow /RecSports/App/
Disallow /RecSports/app/

Comments

  • robots.txt
  • Parasitic bots
  • Aggressive bots that are banned
  • Scirus
  • MJ12bot
  • All robots