Log In Sign Up

venturebeat.com
robots.txt

Robots Exclusion Standard data for venturebeat.com

Resource Scan

Scan Details

Site Domain venturebeat.com
Base Domain venturebeat.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-25T10:59:17+00:00
Next Scan 2024-05-09T10:59:17+00:00

Last Successful Scan

Scanned2024-03-18T10:58:45+00:00
URL https://venturebeat.com/robots.txt
Domain IPs 192.0.66.2
Response IP 192.0.66.2
Found Yes
Hash 974975e20e1e3d6fb97cd508963dc7ff682d09213213f5527e5bf9bc4c429026
SimHash 461f9100b237

Groups

*

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts
Disallow /users

googlebot

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

bingbot

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

Other Records

Field Value
crawl-delay 5

msnbot

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

Other Records

Field Value
crawl-delay 5

bingpreview

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

irlbot

Rule Path
Disallow /login
Disallow /logout
Disallow /sign-up
Disallow /account
Disallow /wp-admin
Disallow /wp-login.php
Disallow /activate
Disallow /search
Disallow /?s=*
Disallow /page/*/?s=*
Disallow /page/*/?s
Disallow /person
Disallow /company
Disallow /posts

motominerbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

df bot 1.0

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

mozilla/5.0 (compatible; neevabot/1.0; +https://neeva.com/neevabot)

Rule Path
Disallow /

awariosmartbot/1.0 (+https://awario.com/bots.html; bots@awario.com)

Rule Path
Disallow /

mozilla/5.0 (compatible; seekport crawler; http://seekport.com/)

Rule Path
Disallow /

mozilla/5.0 (compatible; sphereup; http://www.sphereup.com/)

Rule Path
Disallow /

proximic

Rule Path
Disallow

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://venturebeat.com/news-sitemap.xml
sitemap https://venturebeat.com/sitemap.xml
Back to top

Comments

  • This file was generated on Mon, 18 Mar 2024 10:27:19 +0000
  • Sitemap archive
  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
Back to top