Log In Sign Up

xapiens.id
robots.txt

Robots Exclusion Standard data for xapiens.id

Resource Scan

Scan Details

Site Domain xapiens.id
Base Domain xapiens.id
Scan Status Ok
Last Scan2025-11-12T20:36:59+00:00
Next Scan 2025-12-12T20:36:59+00:00

Last Scan

Scanned2025-11-12T20:36:59+00:00
URL https://xapiens.id/robots.txt
Domain IPs 104.26.0.103, 104.26.1.103, 172.67.75.133, 2606:4700:20::681a:167, 2606:4700:20::681a:67, 2606:4700:20::ac43:4b85
Response IP 104.26.1.103
Found Yes
Hash d774df3f00b39e05a14fb4dc126d97caa64d5a43f4cfb3bf2145d731ac528795
SimHash 447e4d60a636

Groups

*

Rule Path
Allow /
Allow /insights/
Allow /solutions/
Disallow /api/
Disallow /slice-simulator/
Disallow /_app/
Disallow /build/
Disallow /node_modules/
Disallow /.git/
Disallow /.env
Disallow /logs/
Disallow /docker-compose.yml
Disallow /Dockerfile
Disallow /admin/
Disallow /dashboard/
Disallow /*?preview=*
Disallow /api/whatsapp
Disallow /api/whistleblowing
Disallow /*?utm_*
Disallow /*?ref=*
Disallow /*?source=*
Disallow /*?campaign=*

googlebot

Rule Path
Allow /
Allow /insights/
Allow /solutions/

bingbot

Rule Path
Allow /
Allow /insights/
Allow /solutions/

slurp

Rule Path
Allow /
Allow /insights/
Allow /solutions/

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 0.5
Back to top

Other Records

Field Value
sitemap https://xapiens.id/sitemap.xml
Back to top

Comments

  • Robots.txt for Xapiens Teknologi Indonesia
  • https://xapiens.id
  • Allow crawling of main content
  • Block sensitive or unnecessary paths
  • Block admin/preview paths
  • Block form submission endpoints
  • Block query parameters that don't create unique content
  • Allow specific bots with special permissions
  • Block aggressive bots
  • Sitemap location
  • Crawl delay for general bots (0.5 second)
Back to top