Log In Sign Up

phoenix.library.northeastern.edu
robots.txt

Robots Exclusion Standard data for phoenix.library.northeastern.edu

Resource Scan

Scan Details

Site Domain phoenix.library.northeastern.edu
Base Domain northeastern.edu
Scan Status Ok
Last Scan2024-05-31T22:32:59+00:00
Next Scan 2024-06-30T22:32:59+00:00

Last Scan

Scanned2024-05-31T22:32:59+00:00
URL http://phoenix.library.northeastern.edu/robots.txt
Domain IPs 155.33.31.234
Response IP 155.33.31.234
Found Yes
Hash 5a65c6944448633b8fd9708806f8b58f88621de233175b4b2ed16e36452bbe70
SimHash 43fcdf61c680

Groups

*

Rule Path
Disallow /wp-content/uploads/
Disallow /wp-content/plugins/
Disallow /readme.html
Disallow /boston/
Disallow /supplements/
Disallow /wfnx-fm/
Disallow /archive/
Disallow /listings/

Other Records

Field Value
crawl-delay 190

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yak

Rule Path
Disallow /

clikzbot

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/fast/2.0

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

owlin bot

Rule Path
Disallow /

gnowitnewsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

wazzup

Rule Path
Disallow /

live36.ru

Rule Path
Disallow /

vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the crawl-delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

megaindex.ru

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

Rule Path
Disallow /

cutbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

pingbot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://phoenix.library.northeastern.edu/sitemap.xml
Back to top

Comments

  • ADDED for time being
  • Disallow: /
  • Baiduspider
  • Yandex
  • User-Agent: *
  • ADDED for time being
  • Disallow: /
Back to top