Log In Sign Up

sematext.com
robots.txt

Robots Exclusion Standard data for sematext.com

Resource Scan

Scan Details

Site Domain sematext.com
Base Domain sematext.com
Scan Status Ok
Last Scan2024-09-21T12:57:34+00:00
Next Scan 2024-09-28T12:57:34+00:00

Last Scan

Scanned2024-09-21T12:57:34+00:00
URL https://sematext.com/robots.txt
Domain IPs 52.70.188.24, 54.82.214.135
Response IP 52.70.188.24
Found Yes
Hash d9ad264a4fd44e0d782d82421937db87819a2b785eb8e936daf6d62a599e1934
SimHash 5d204e3b8792

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /readme.html
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Disallow /docs/Monitoring/
Disallow /docs/Logs/
Disallow /docs/Logagent/
Disallow /docs/Sematext-Enterprise/
Allow /wp-admin/admin-ajax.php
Disallow /*.pdf$
Disallow /lp/
Disallow /opensee/api
Back to top

Other Records

Field Value
sitemap https://sematext.com/sitemap.xml
sitemap https://sematext.com/integrations-sitemap/
Back to top

Comments

  • Disallow: /opensee/jd
  • Disallow: /opensee/javadoc
  • Disallow: /opensee/c
  • Disallow: /opensee/m/*/*/plain
  • Don't index URLs with question marks
  • Noindex: /opensee/*?*
  • Disallow: /opensee/*?*
  • https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/
  • Noindex: /opensee/m/
  • Disallow: /opensee/m/
  • Noindex: /opensee/jd/
  • Disallow: /opensee/jd/
  • Noindex: /opensee/javadoc/
  • Disallow: /opensee/javadoc/
  • Noindex: /opensee/c/
  • Disallow: /opensee/c/
  • Disallow: /opensee/report/author/
  • Disallow: /opensee/report/*/author/
Back to top