Log In Sign Up

randomlygenerate.com
robots.txt

Robots Exclusion Standard data for randomlygenerate.com

Resource Scan

Scan Details

Site Domain randomlygenerate.com
Base Domain randomlygenerate.com
Scan Status Ok
Last Scan2025-09-16T23:57:32+00:00
Next Scan 2025-10-16T23:57:32+00:00

Last Scan

Scanned2025-09-16T23:57:32+00:00
URL https://randomlygenerate.com/robots.txt
Redirect https://www.randomlygenerate.com/robots.txt
Redirect Domain www.randomlygenerate.com
Redirect Base randomlygenerate.com
Domain IPs 104.21.80.73, 172.67.175.225, 2606:4700:3031::ac43:afe1, 2606:4700:3034::6815:5049
Redirect IPs 104.21.80.73, 172.67.175.225, 2606:4700:3031::ac43:afe1, 2606:4700:3034::6815:5049
Response IP 104.21.80.73
Found Yes
Hash 6a1c091d12ebbea526125cc8cd4618afa6327c54f29424b02b50eb3b19876a8d
SimHash 63068a52e6a5

Groups

*

Rule Path
Allow /
Disallow /assets/js/
Disallow /assets/css/
Disallow /workers/
Disallow /admin/
Disallow /private/
Allow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://randomlygenerate.com/sitemap.xml
Back to top

Comments

  • Block access to sensitive directories
  • Allow access to all HTML pages and important assets
  • Sitemap location
  • Crawl delay for polite crawling
Back to top