Log In Sign Up

participa.gencat.cat
robots.txt

Robots Exclusion Standard data for participa.gencat.cat

Resource Scan

Scan Details

Site Domain participa.gencat.cat
Base Domain gencat.cat
Scan Status Ok
Last Scan2024-05-23T13:49:43+00:00
Next Scan 2024-06-22T13:49:43+00:00

Last Scan

Scanned2024-05-23T13:49:43+00:00
URL https://participa.gencat.cat/robots.txt
Domain IPs 23.97.231.17
Response IP 23.97.231.17
Found Yes
Hash e0584e9556ee235660dd394b0e788f331370422a9659f05af45b24d729a33b3b
SimHash b8801f0f68e0

Groups

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-seoab

Rule Path
Disallow /

search.msn.com

Rule Path
Disallow /

msnbot

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To block SEMrushBot from crawling your site for different SEO and technical issues:
  • To block SEMrushBot from crawling your site for Backlink Audit tool:
  • To block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SEMrushBot from checking URLs on your site for SWA tool:
  • To block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SEMrushBot from crawling your site for Brand Monitoring:
  • To block SEMrushBot from crawling your site for SEO A/B Testing tool:
  • msnbot-13-66-139-20.search.msn.com.
  • User-agent: bingbot
  • Disallow: /
Back to top