Log In Sign Up

cname.bitly.com
robots.txt

Robots Exclusion Standard data for cname.bitly.com

Resource Scan

Scan Details

Site Domain cname.bitly.com
Base Domain bitly.com
Scan Status Ok
Last Scan2024-04-25T11:48:56+00:00
Next Scan 2024-05-25T11:48:56+00:00

Last Scan

Scanned2024-04-25T11:48:56+00:00
URL http://cname.bitly.com/robots.txt
Domain IPs 67.199.248.12, 67.199.248.13
Response IP 67.199.248.12
Found Yes
Hash d91cb4a1f2ea14ba0b6bcb4e1f692f767097ff7b35cb3c1061097497efdd12b7
SimHash 261cc1c9edd3

Groups

*

Rule Path
Disallow /pages/landing$
Disallow /pages/es$
Disallow /pages/pt$
Disallow /pages/it$
Disallow /pages/de$
Disallow /pages/fr$
Disallow /pages/home
Disallow /pages/thank-you
Disallow /pages/free-plan-refresh-faqs
Disallow /blog/*?
Disallow /blog/?p=
Disallow /blog/page/*/?p=
Disallow /pages/*?preview%5B
Disallow /pages/*%26preview%5B
Disallow /pages/*%5B
Disallow /pages/*%7B
Disallow /pages/*%3B
Back to top

Other Records

Field Value
sitemap https://bitly.com/pages/sitemap_index.xml
sitemap https://bitly.com/blog/sitemap_index.xml
Back to top

Comments

  • Welcome to Bitly =)
  • robots welcome;
  • API documentation can be found at https://dev.bitly.com/
Back to top