Log In Sign Up

scaler.com
robots.txt

Robots Exclusion Standard data for scaler.com

Resource Scan

Scan Details

Site Domain scaler.com
Base Domain scaler.com
Scan Status Ok
Last Scan2024-11-11T02:55:23+00:00
Next Scan 2024-11-18T02:55:23+00:00

Last Scan

Scanned2024-11-11T02:55:23+00:00
URL https://scaler.com/robots.txt
Redirect https://www.scaler.com/robots.txt
Redirect Domain www.scaler.com
Redirect Base scaler.com
Domain IPs 13.226.61.111, 13.226.61.125, 13.226.61.21, 13.226.61.94
Redirect IPs 13.33.88.127, 13.33.88.32, 13.33.88.33, 13.33.88.43
Response IP 13.33.88.33
Found Yes
Hash b9fd2e0f3fccfc9a336211ee49fd80e9a7bc275acb188a63897fbca0458a61b5
SimHash b2848d052d70

Groups

*

Rule Path
Disallow /profile
Disallow /event/*
Disallow /topics/*?
Disallow /topics/page/*
Disallow /topics/category/*
Disallow /topics/*/amp/*
Disallow /academy/mentor/profile/
Disallow /academy/profile/
Disallow /careers/
Allow /careers/$
Disallow /users/sign_up/
Disallow /users/confirmation/new/
Disallow /users/password/new/
Disallow /users/sign_in/mobile/
Back to top

Other Records

Field Value
sitemap https://www.scaler.com/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top