Log In Sign Up

twittter.com
robots.txt

Robots Exclusion Standard data for twittter.com

Resource Scan

Scan Details

Site Domain twittter.com
Base Domain twittter.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-11-10T18:49:09+00:00
Next Scan 2026-02-08T18:49:09+00:00

Last Successful Scan

Scanned2022-12-30T08:58:12+00:00
URL http://twittter.com/robots.txt
Redirect https://twitter.com/robots.txt
Redirect Domain twitter.com
Redirect Base twitter.com
Domain IPs 199.59.148.10, 199.59.148.82, 199.59.150.39, 199.59.150.7
Redirect IPs 104.244.42.193, 104.244.42.65
Response IP 104.244.42.129
Found Yes
Hash a190f80b39dcbfd22521f33a2e1e953b568b949f71ce727644e00de79a285e5e
SimHash 2e7efa11e5f5

Groups

googlebot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Allow /*?ref_src=
Allow /*?src=
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

slurp

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

yandex

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

msnbot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

bingbot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

*

Rule Path
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated
Disallow /oauth
Disallow /1/oauth
Disallow /i/streams
Disallow /i/hello
Disallow /i/u

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://twitter.com/sitemap.xml
Back to top

Comments

  • Google Search Engine Robot
  • ==========================
  • Yahoo! Search Engine Robot
  • ==========================
  • Yandex Search Engine Robot
  • ==========================
  • Microsoft Search Engine Robot
  • =============================
  • Bing Search Engine Robot
  • ========================
  • Every bot that might possibly read and respect this file
  • ========================================================
  • WHAT-4882 - Block indexing of links in notification emails. This applies to all bots.
  • =====================================================================================
  • Wait 1 second between successive requests. See ONBOARD-2698 for details.
  • Independent of user agent. Links in the sitemap are full URLs using https:// and need to match
  • the protocol of the sitemap.
Back to top

Warnings

  • `noindex` is not a known field.
Back to top