Log In Sign Up

bitly.com
robots.txt

Robots Exclusion Standard data for bitly.com

Resource Scan

Scan Details

Site Domain bitly.com
Base Domain bitly.com
Scan Status Ok
Last Scan2024-11-02T10:27:43+00:00
Next Scan 2024-11-16T10:27:43+00:00

Last Scan

Scanned2024-11-02T10:27:43+00:00
URL https://bitly.com/robots.txt
Domain IPs 67.199.248.14, 67.199.248.15
Response IP 67.199.248.15
Found Yes
Hash 0c0e82c343d175553db8658a4066e358a7e2618d2fa39ef2c51703195f9b250c
SimHash 231cc54aedd3

Groups

*

Rule Path
Disallow /pages/landing
Disallow /pages/pt$
Disallow /pages/home
Disallow /pages/thank-you
Disallow /pages/demo-thank-you
Disallow /pages/email-thank-you
Disallow /pages/free-plan-refresh-faqs
Disallow /pages/*/v1$
Disallow /pages/*/v2$
Disallow /blog/*?
Disallow /blog/?p=
Disallow /blog/page/*/?p=
Disallow /pages/*?preview%5B
Disallow /pages/*%26preview%5B
Disallow /pages/*%5B
Disallow /pages/*%7B
Disallow /pages/*%3B
Back to top

Other Records

Field Value
sitemap https://bitly.com/pages/sitemap_index.xml
sitemap https://bitly.com/blog/sitemap_index.xml
Back to top

Comments

  • Welcome to Bitly =)
  • robots welcome;
  • API documentation can be found at https://dev.bitly.com/
Back to top