Log In Sign Up

bxscience.edu
robots.txt

Robots Exclusion Standard data for bxscience.edu

Resource Scan

Scan Details

Site Domain bxscience.edu
Base Domain bxscience.edu
Scan Status Ok
Last Scan2024-08-29T12:38:45+00:00
Next Scan 2024-09-28T12:38:45+00:00

Last Scan

Scanned2024-08-29T12:38:45+00:00
URL https://bxscience.edu/robots.txt
Domain IPs 34.85.181.171
Response IP 34.85.181.171
Found Yes
Hash 0d646592747ab5cbb6ec15471fb2e8e3f3ba062b21d7747e110ed49ac93fd15b
SimHash 73dd9d510eb9

Groups

googlebot

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

googlebot-mobile

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

w3c-checklink

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/
Disallow /m/classes/
Disallow /ourpages

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

msnbot

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

accessiblewebbot

Rule Path
Disallow /apps/ajax/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/temp.jsp
Disallow /m/api

facebookexternalhit

Rule Path
Disallow /apps/ajax/
Disallow /apps/classes/
Disallow /apps/directory/
Disallow /apps/events/events_rss.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/view_calendar.jsp
Disallow /apps/events/list_pff.jsp
Disallow /apps/events/calendar_yearly_pff.jsp
Disallow /apps/events/calendar_pff.jsp
Disallow /apps/events/calendar_month_pff.jsp
Disallow /apps/email
Disallow /apps/email/
Disallow /apps/email/index.jsp
Disallow /apps/file_manager/
Disallow /apps/mediaplayer-3-16/
Disallow /apps/ourpages/
Disallow /apps/ourpages
Disallow /apps/forms/
Disallow /apps/login/
Disallow /apps/s/
Disallow /apps/thickbox/
Disallow /apps/tools/
Disallow /apps/videos/
Disallow /apps/cross.jsp
Disallow /apps/temp.jsp
Disallow /m/api
Disallow /m/classes

Other Records

Field Value
crawl-delay 1

rogerbot

Rule Path
Disallow

dotbot

Rule Path
Disallow

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

scrapybot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

powermapper

Rule Path
Allow /

*

Rule Path
Disallow /
Allow /robots.txt
Back to top

Other Records

Field Value
sitemap https://bxscience.edu/sitemap.xml
Back to top