astro2sphere.com
robots.txt

Robots Exclusion Standard data for astro2sphere.com

Resource Scan

Scan Details

Site Domain astro2sphere.com
Base Domain astro2sphere.com
Scan Status Ok
Last Scan2025-10-26T19:50:01+00:00
Next Scan 2025-11-25T19:50:01+00:00

Last Scan

Scanned2025-10-26T19:50:01+00:00
URL https://astro2sphere.com/robots.txt
Redirect https://www.astro2sphere.com/robots.txt
Redirect Domain www.astro2sphere.com
Redirect Base astro2sphere.com
Domain IPs 216.198.79.1
Redirect IPs 216.198.79.1, 64.29.17.1
Response IP 64.29.17.65
Found Yes
Hash 11fda648d8b1771c010b72317e5c81ef5d1d25a20dd7f5da46ca2892f3b10da8
SimHash 6419dcb0e4f5

Groups

*

Rule Path
Allow /
Allow /about
Allow /services
Allow /astro-blogs
Allow /contact
Allow /booking
Disallow /admin/
Disallow /user/
Disallow /api/
Disallow /login
Disallow /signup
Disallow /forgot-password
Disallow /dashboard/
Disallow /profile/
Disallow /my-appointments/
Allow /services
Allow /contact
Allow /about
Allow /astro-blogs
Allow /book-appointment

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://astro2sphere.com/sitemap.xml

Comments

  • Astro2Sphere Official Website
  • https://astro2sphere.com
  • High priority pages for Astro2Sphere brand
  • Block access to admin and user dashboard areas
  • Block access to authentication pages for bots
  • Block access to private user content
  • Sitemap location
  • Crawl-delay for respectful crawling
  • Specific rules for major search engines
  • Block known bad bots