Log In Sign Up

devcrew.io
robots.txt

Robots Exclusion Standard data for devcrew.io

Resource Scan

Scan Details

Site Domain devcrew.io
Base Domain devcrew.io
Scan Status Ok
Last Scan2025-09-21T15:26:36+00:00
Next Scan 2025-10-21T15:26:36+00:00

Last Scan

Scanned2025-09-21T15:26:36+00:00
URL https://devcrew.io/robots.txt
Domain IPs 104.21.5.171, 172.67.133.168, 2606:4700:3030::ac43:85a8, 2606:4700:3033::6815:5ab
Response IP 104.21.5.171
Found Yes
Hash 7ce6b190b3f02c3ae342f2a6d988e8493d43bfa1af73380db0f5aa994a4dfded
SimHash 6140515102f0

Groups

*

Rule Path
Disallow /readme.html
Disallow /refer/
Disallow /wp-includes/
Disallow /wp-content/cache/
Disallow /trackback/
Disallow /*.sql$
Disallow /*.tgz$
Disallow /*.php$
Disallow /*.inc$
Disallow /*.gz$
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /new-login/
Allow /wp-admin/admin-ajax.php
Back to top

Other Records

Field Value
sitemap https://devcrew.io/post-sitemap.xml
sitemap https://devcrew.io/page-sitemap.xml
sitemap https://devcrew.io/project-sitemap.xml
sitemap https://devcrew.io/category-sitemap.xml
Back to top

Comments

  • robots.txt for devcrew.io
  • Last updated: 2025-07-24
  • Sitemaps
  • Block unnecessary files and folders
  • Block WordPress admin and login pages (allow AJAX)
  • End of file
Back to top