Log In Sign Up

prometheus.systems
robots.txt

Robots Exclusion Standard data for prometheus.systems

Resource Scan

Scan Details

Site Domain prometheus.systems
Base Domain prometheus.systems
Scan Status Ok
Last Scan2024-10-01T18:46:42+00:00
Next Scan 2024-10-31T18:46:42+00:00

Last Scan

Scanned2024-10-01T18:46:42+00:00
URL https://prometheus.systems/robots.txt
Domain IPs 216.249.100.84
Response IP 216.249.100.84
Found Yes
Hash fa7907a8180957167ac51ecc23db7f34fe5bcaa481526c3007fbf9063d3fa0c5
SimHash a9246f33cf26

Groups

*

Rule Path
Disallow /internal/
Allow /

*

Rule Path
Allow /posts/
Allow /store/
Back to top

Other Records

Field Value
sitemap https://prometheus.systems/sitemap.xml
sitemap https://prometheus.systems/posts/sitemap.xml
sitemap https://prometheus.systems/store/sitemap.xml
Back to top

Comments

  • Allow all user-agents to crawl the main site, except pages for internal use
  • Allow crawling of WordPress sites
  • Sitemap locations
Back to top