twitter.com.br
robots.txt

Robots Exclusion Standard data for twitter.com.br

Resource Scan

Scan Details

Site Domain twitter.com.br
Base Domain twitter.com.br
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-03-22T17:05:18+00:00
Next Scan 2024-06-20T17:05:18+00:00

Last Successful Scan

Scanned2023-05-01T21:34:06+00:00
URL http://www.twitter.com.br/robots.txt
Redirect https://twitter.com/robots.txt
Redirect Domain twitter.com
Redirect Base twitter.com
Domain IPs 104.244.42.1, 104.244.42.129, 104.244.42.193, 104.244.42.65
Redirect IPs 104.244.42.1, 104.244.42.129, 104.244.42.193, 104.244.42.65
Response IP 104.244.42.193
Found Yes
Hash a190f80b39dcbfd22521f33a2e1e953b568b949f71ce727644e00de79a285e5e
SimHash 2e7efa11e5f5

Groups

googlebot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Allow /*?ref_src=
Allow /*?src=
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

slurp

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

yandex

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

msnbot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

bingbot

Rule Path
Allow /?_escaped_fragment_
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated

*

Rule Path
Allow /*?lang=
Allow /hashtag/*?src=
Allow /search?q=%23
Allow /i/api/
Disallow /search/realtime
Disallow /search/users
Disallow /search/*/grid
Disallow /*?
Disallow /*/followers
Disallow /*/following
Disallow /account/deactivated
Disallow /settings/deactivated
Disallow /oauth
Disallow /1/oauth
Disallow /i/streams
Disallow /i/hello
Disallow /i/u

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://twitter.com/sitemap.xml

Comments

  • Google Search Engine Robot
  • ==========================
  • Yahoo! Search Engine Robot
  • ==========================
  • Yandex Search Engine Robot
  • ==========================
  • Microsoft Search Engine Robot
  • =============================
  • Bing Search Engine Robot
  • ========================
  • Every bot that might possibly read and respect this file
  • ========================================================
  • WHAT-4882 - Block indexing of links in notification emails. This applies to all bots.
  • =====================================================================================
  • Wait 1 second between successive requests. See ONBOARD-2698 for details.
  • Independent of user agent. Links in the sitemap are full URLs using https:// and need to match
  • the protocol of the sitemap.

Warnings

  • `noindex` is not a known field.