Log In Sign Up

thebroadcastbridge.com
robots.txt

Robots Exclusion Standard data for thebroadcastbridge.com

Resource Scan

Scan Details

Site Domain thebroadcastbridge.com
Base Domain thebroadcastbridge.com
Scan Status Ok
Last Scan2024-09-21T16:53:53+00:00
Next Scan 2024-10-21T16:53:53+00:00

Last Scan

Scanned2024-09-21T16:53:53+00:00
URL https://thebroadcastbridge.com/robots.txt
Redirect https://www.thebroadcastbridge.com/robots.txt
Redirect Domain www.thebroadcastbridge.com
Redirect Base thebroadcastbridge.com
Domain IPs 104.26.0.104, 104.26.1.104, 172.67.74.28, 2606:4700:20::681a:168, 2606:4700:20::681a:68, 2606:4700:20::ac43:4a1c
Redirect IPs 104.26.0.104, 104.26.1.104, 172.67.74.28, 2606:4700:20::681a:168, 2606:4700:20::681a:68, 2606:4700:20::ac43:4a1c
Response IP 104.26.0.104
Found Yes
Hash 9271d972ef012dafc7a00454a0d81f9e62beec53c1a991097edf32077ecb0d37
SimHash 4256ca53f65a

Groups

*

Rule Path
Disallow /documentation/
Disallow /tmp/
Disallow /member/
Disallow /dictionary
Disallow /uploads/
Disallow /digest/pending
Disallow /sandbox
Disallow /email
Disallow /cache/uploads/

Other Records

Field Value
crawl-delay 3

swiftbot

Rule Path
Disallow

voltron

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

adbeat_bot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bubing

Rule Path
Disallow /

botify

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.thebroadcastbridge.com/site-map/google
Back to top

Comments

  • https://en.wikipedia.org/wiki/Robots_exclusion_standard
  • bingbot/2.0
  • Googlebot/2.1
  • AddThis.com
  • WOW64 https://www.bing.com/webmaster/help/how-to-use-the-verify-bingbot-tool-2195837e
  • setup webmaster rules for BING and googles
  • https://www.bing.com/webmaster/help/crawl-control-55a30302
  • Block common cralwers:
  • Some weird backlink checker (webmeup.com)
  • Another backlink crawler
  • Another backlink crawler (https://ahrefs.com/robot)
  • Another one.
  • 2017/05/23 bot hit site and slowed.
  • 2017/05/23 bot hit site and slowed.
  • 2017/09/05 bot hit site and slowed.
  • Block trendkite-akashic-crawler
Back to top