Log In Sign Up

angellist.com
robots.txt

Robots Exclusion Standard data for angellist.com

Resource Scan

Scan Details

Site Domain angellist.com
Base Domain angellist.com
Scan Status Ok
Last Scan2024-06-07T16:31:15+00:00
Next Scan 2024-07-07T16:31:15+00:00

Last Scan

Scanned2024-06-07T16:31:15+00:00
URL https://angellist.com/robots.txt
Redirect https://www.angellist.com/robots.txt
Redirect Domain www.angellist.com
Redirect Base angellist.com
Domain IPs 104.22.72.200, 104.22.73.200, 172.67.20.123, 2606:4700:10::6816:48c8, 2606:4700:10::6816:49c8, 2606:4700:10::ac43:147b
Redirect IPs 76.76.21.241, 76.76.21.61
Response IP 76.76.21.241
Found Yes
Hash dce2540b2ddd941490969b68f391b3b7992966407c2904f754824f11c63b7a8d
SimHash c454cb29f600

Groups

voltron

Rule Path
Disallow /

feedburner/1.0 (http://www.feedburner.com)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /
Allow /api/og/*
Back to top

Comments

  • Somehow being used to fetch user profiles?
  • Some weird backlink checker (webmeup.com)
  • Another backlink crawler
  • Another one.
Back to top