Log In Sign Up

participa.gencat.cat
robots.txt

Robots Exclusion Standard data for participa.gencat.cat

Resource Scan

Scan Details

Site Domain participa.gencat.cat
Base Domain gencat.cat
Scan Status Ok
Last Scan2024-10-20T16:41:16+00:00
Next Scan 2024-11-19T16:41:16+00:00

Last Scan

Scanned2024-10-20T16:41:16+00:00
URL https://participa.gencat.cat/robots.txt
Domain IPs 4.245.83.31
Response IP 4.245.83.31
Found Yes
Hash ec63426c9bd4e4fe3a950861b607cc7687e90e993653355875acb69870e2d656
SimHash b8801f0be8e0

Groups

*

Rule Path
Disallow /profiles/
Disallow /search

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-seoab

Rule Path
Disallow /

search.msn.com

Rule Path
Disallow /

msnbot

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To block SEMrushBot from crawling your site for different SEO and technical issues:
  • To block SEMrushBot from crawling your site for Backlink Audit tool:
  • To block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SEMrushBot from checking URLs on your site for SWA tool:
  • To block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SEMrushBot from crawling your site for Brand Monitoring:
  • To block SEMrushBot from crawling your site for SEO A/B Testing tool:
  • msnbot-13-66-139-20.search.msn.com.
  • User-agent: bingbot
  • Disallow: /
Back to top