botio.io
robots.txt

Robots Exclusion Standard data for botio.io

Resource Scan

Scan Details

Site Domain botio.io
Base Domain botio.io
Scan Status Ok
Last Scan2025-10-05T20:38:40+00:00
Next Scan 2025-10-19T20:38:40+00:00

Last Scan

Scanned2025-10-05T20:38:40+00:00
URL https://botio.io/robots.txt
Domain IPs 104.21.73.47, 172.67.140.150, 2606:4700:3035::ac43:8c96, 2606:4700:3036::6815:492f
Response IP 172.67.140.150
Found Yes
Hash d4be3da0eaef5d010e2483f9d054e0f0db95b638e05d4b6315cbc40e8ff3774d
SimHash 680cc643f7b8

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /blog/admin/
Disallow /api/
Allow /blog/
Allow /voucher-launcher
Allow /commerce-launcher
Allow /reward-launcher

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

googlebot-image

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://botio.io/sitemap.xml
sitemap https://botio.io/api/sitemap.xml

Comments

  • Sitemap locations
  • Crawl-delay for polite crawling
  • Disallow admin and private areas
  • Allow important directories
  • Allow common bots
  • Google Image indexing
  • Block bad bots