Log In Sign Up

skybots.tech
robots.txt

Robots Exclusion Standard data for skybots.tech

Resource Scan

Scan Details

Site Domain skybots.tech
Base Domain skybots.tech
Scan Status Ok
Last Scan2025-10-20T17:48:26+00:00
Next Scan 2025-10-27T17:48:26+00:00

Last Scan

Scanned2025-10-20T17:48:26+00:00
URL https://skybots.tech/robots.txt
Domain IPs 104.21.21.76, 172.67.197.8, 2606:4700:3031::6815:154c, 2606:4700:3037::ac43:c508
Response IP 172.67.197.8
Found Yes
Hash 16b62edb693946211680284d6a752a66b3132ff089d3dbbd78a706c7d04020f5
SimHash 4a1ebc0367f0

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /dashboard/admin/
Disallow /admin/
Disallow /admin-emergency/
Disallow /_next/static/chunks/
Disallow /auth-debug
Disallow /auth-error
Disallow /banned

Other Records

Field Value
crawl-delay 2

googlebot

Rule Path
Allow /
Allow /_next/static/css/
Allow /_next/static/js/
Allow /images/
Allow /logo/
Allow /flags/
Disallow /api/
Disallow /dashboard/admin/
Disallow /admin/
Disallow /admin-emergency/
Disallow /_next/static/chunks/
Disallow /auth-debug
Disallow /auth-error
Disallow /banned

bingbot

Rule Path
Allow /
Allow /_next/static/css/
Allow /_next/static/js/
Allow /images/
Allow /logo/
Allow /flags/
Disallow /api/
Disallow /dashboard/admin/
Disallow /admin/
Disallow /admin-emergency/
Disallow /_next/static/chunks/
Disallow /auth-debug
Disallow /auth-error
Disallow /banned

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /
Disallow /api/
Disallow /dashboard/admin/

Other Records

Field Value
crawl-delay 2

facebookexternalhit

Rule Path
Allow /
Allow /images/
Allow /logo/

twitterbot

Rule Path
Allow /
Allow /images/
Allow /logo/
Back to top

Other Records

Field Value
sitemap https://skybots.tech/sitemap.xml
Back to top

Comments

  • Sitemap
  • Optimized for Googlebot (no crawl-delay - Google ignores it)
  • Optimized for Bingbot
  • Other search engines
  • Social media crawlers (no crawl-delay needed)
Back to top