Log In Sign Up

baylor.edu
robots.txt

Robots Exclusion Standard data for baylor.edu

Resource Scan

Scan Details

Site Domain baylor.edu
Base Domain baylor.edu
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-25T03:25:58+00:00
Next Scan 2024-10-23T03:25:58+00:00

Last Successful Scan

Scanned2023-12-05T22:42:00+00:00
URL https://baylor.edu/robots.txt
Redirect https://www.baylor.edu/robots.txt
Redirect Domain www.baylor.edu
Redirect Base baylor.edu
Domain IPs 129.62.3.230
Redirect IPs 104.16.61.32, 104.16.62.32, 2606:4700::6810:3d20, 2606:4700::6810:3e20
Response IP 104.16.62.32
Found Yes
Hash b07ac73dc8b7810d9bc79d00fe9a8c178e278b961627b47192fd27d90b61b136
SimHash aa152903cf50

Groups

*

Rule Path
Disallow /old/*

semrushbot

Rule Path
Disallow /calendar/*
Back to top

Comments

  • Baylor University
  • This is a file retrieved by webwalkers a.k.a. spiders that
  • conform to a defacto standard.
  • See <URL:http://www.robotstxt.org/wc/exclusion.html#robotstxt>
  • Format is:
  • User-agent: <name of spider>
  • Disallow: <nothing> | <path>
  • -----------------------------------------------------------------------------
Back to top