bostondevices.lk
robots.txt

Robots Exclusion Standard data for bostondevices.lk

Resource Scan

Scan Details

Site Domain bostondevices.lk
Base Domain bostondevices.lk
Scan Status Ok
Last Scan2025-11-12T12:36:35+00:00
Next Scan 2025-12-12T12:36:35+00:00

Last Scan

Scanned2025-11-12T12:36:35+00:00
URL https://bostondevices.lk/robots.txt
Domain IPs 104.21.88.234, 172.67.154.53, 2606:4700:3034::6815:58ea, 2606:4700:3037::ac43:9a35
Response IP 172.67.154.53
Found Yes
Hash 435f9bd5b9878a3c853669f48dcc3c94b8e66236d6c5434e3c8fc067f3a83d42
SimHash 4b3a4861e6b3

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /cgi-bin/
Disallow /xmlrpc.php
Allow /wp-admin/admin-ajax.php

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

facebot

Rule Path
Disallow

facebookexternalhit

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

twitterbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.bostondevices.lk/sitemap_index.xml

Comments

  • robots.txt for https://www.bostondevices.lk
  • Block all bots by default
  • Block spammy crawlers
  • Allow good bots
  • Sitemap