Log In Sign Up

wsubookie.net
robots.txt

Robots Exclusion Standard data for wsubookie.net

Resource Scan

Scan Details

Site Domain wsubookie.net
Base Domain wsubookie.net
Scan Status Ok
Last Scan2025-11-24T20:17:00+00:00
Next Scan 2025-12-24T20:17:00+00:00

Last Scan

Scanned2025-11-24T20:17:00+00:00
URL http://wsubookie.net/robots.txt
Redirect https://wsubookie.bncollege.com/robots.txt
Redirect Domain wsubookie.bncollege.com
Redirect Base bncollege.com
Domain IPs 64.126.133.29
Redirect IPs 104.69.37.80
Response IP 23.50.95.89
Found Yes
Hash 29855c492f3050ab80970c1b33431ff1d9e85270f6b1e513daa9f36851b04dc2
SimHash 7874179ccfe0

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

duckduckbot/1.1

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Request-rate: 1/10 # maximum rate is one page every 10 seconds
  • Crawl-delay: 10 # 10 seconds between page requests
  • Visit-time: 0400-0845 # only visit between 04:00 and 08:45 UTC
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top