Log In Sign Up

betanet.in
robots.txt

Robots Exclusion Standard data for betanet.in

Resource Scan

Scan Details

Site Domain betanet.in
Base Domain betanet.in
Scan Status Ok
Last Scan2025-05-04T11:02:20+00:00
Next Scan 2025-06-03T11:02:20+00:00

Last Scan

Scanned2025-05-04T11:02:20+00:00
URL https://betanet.in/robots.txt
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.80.1
Found Yes
Hash 3779b4823578cbe76314e86fbe2f5c2f68890119ee024359e29de36ed05ca8bf
SimHash 6c18588167cd

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /cgi-bin/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /?s=
Disallow /comments/
Disallow /feed/

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Disallow /

slurp

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

slurp

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

curl

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

wget

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

webcopier

Rule Path
Disallow /
Back to top

Comments

  • =====================
  • General Bot Controls
  • =====================
  • Optional: Block comment and feed URLs (can be uncommented if needed)
  • Disallow: /category/
  • Disallow: /tag/
  • =====================
  • Major Search Engine Bots - Allow with Crawl Delay
  • =====================
  • =====================
  • Sitemap (Replace Per Site or Remove)
  • =====================
  • Sitemap: https://yourdomain.com/sitemap.xml
Back to top