Log In Sign Up

rutgers.bncollege.com
robots.txt

Robots Exclusion Standard data for rutgers.bncollege.com

Resource Scan

Scan Details

Site Domain rutgers.bncollege.com
Base Domain bncollege.com
Scan Status Ok
Last Scan2025-09-29T21:24:59+00:00
Next Scan 2025-10-29T21:24:59+00:00

Last Scan

Scanned2025-09-29T21:24:59+00:00
URL https://rutgers.bncollege.com/robots.txt
Domain IPs 23.39.10.236
Response IP 23.39.10.236
Found Yes
Hash 29855c492f3050ab80970c1b33431ff1d9e85270f6b1e513daa9f36851b04dc2
SimHash 7874179ccfe0

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

duckduckbot/1.1

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Request-rate: 1/10 # maximum rate is one page every 10 seconds
  • Crawl-delay: 10 # 10 seconds between page requests
  • Visit-time: 0400-0845 # only visit between 04:00 and 08:45 UTC
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top