Log In Sign Up

codearistos.net
robots.txt

Robots Exclusion Standard data for codearistos.net

Resource Scan

Scan Details

Site Domain codearistos.net
Base Domain codearistos.net
Scan Status Ok
Last Scan2025-07-16T14:50:22+00:00
Next Scan 2025-08-15T14:50:22+00:00

Last Scan

Scanned2025-07-16T14:50:22+00:00
URL https://codearistos.net/robots.txt
Redirect https://www.codearistos.net/robots.txt
Redirect Domain www.codearistos.net
Redirect Base codearistos.net
Domain IPs 64.29.17.1, 64.29.17.65
Redirect IPs 216.198.79.1, 216.198.79.65
Response IP 216.198.79.65
Found Yes
Hash 2f58388b7a1bc7f5b499f4b62327b9916bc36aa01d3922b80ea2e0e87cb4e1a2
SimHash 681cf092eee1

Groups

*

Rule Path Comment
Allow / -
Allow /products/ -
Allow /about/ -
Allow /contact/ -
Disallow /admin/ -
Disallow /customer/ -
Disallow /api/ -
Disallow /private/ -
Disallow /*?* Block URLs with query parameters
Disallow /*.json$ -
Disallow /*.xml$ -
Disallow /*.php$ -

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Allow all crawlers
  • Allow access to most pages
  • Prevent access to admin and private areas
  • Prevent access to specific file types
  • Add sitemap location (uncomment once sitemap is created)
  • Sitemap: https://your-domain.com/sitemap.xml
  • Crawl-delay for all bots
Back to top