Log In Sign Up

digitalpoint.com
robots.txt

Robots Exclusion Standard data for digitalpoint.com

Resource Scan

Scan Details

Site Domain digitalpoint.com
Base Domain digitalpoint.com
Scan Status Ok
Last Scan2024-06-29T12:36:42+00:00
Next Scan 2024-07-06T12:36:42+00:00

Last Scan

Scanned2024-06-29T12:36:42+00:00
URL https://www.digitalpoint.com/robots.txt
Domain IPs 104.26.12.220, 104.26.13.220, 172.67.72.172, 2606:4700:20::681a:cdc, 2606:4700:20::681a:ddc, 2606:4700:20::ac43:48ac
Response IP 104.26.12.220
Found Yes
Hash deb7558ad2d82ed54263e2d5b28af97335557edf4f541c82d3a515657d591193
SimHash 063cd932d611

Groups

mediapartners-google

Rule Path
Allow /conversations/
Allow /account/

*

Rule Path
Disallow /account/
Disallow /conversations/
Disallow /find-new/
Disallow /login
Disallow /posts/*/ip$
Disallow /posts/*/tweet
Disallow /members/*/trophies
Disallow /search/

baiduspider

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • User-agent: bingbot
  • going to lift it's crawl delay restrictions now that they moved to HTTP/1.1
  • Crawl-delay: 10
  • /sbin/route add -net 65.52.0.0 netmask 255.252.0.0 reject // block all of Microsoft if it doesn't adhere
  • Does not use HTTP/1.1 with compression
  • These are the brain dead spiders from major search engines
  • User-agent: Yahoo! Slurp
  • Disallow: /
  • Allow: if ($_SERVER["SERVER_PROTOCOL"] === 'HTTP/1.1' || $relevancy > 0)
  • Learn to use a HTTP protocol standard that's more than a decade old
  • http://www.w3.org/Protocols/rfc2068/rfc2068.txt
Back to top