insta-market.de
robots.txt

Robots Exclusion Standard data for insta-market.de

Resource Scan

Scan Details

Site Domain insta-market.de
Base Domain insta-market.de
Scan Status Ok
Last Scan5/10/2025, 1:06:23 PM
Next Scan 6/9/2025, 1:06:23 PM

Last Scan

Scanned5/10/2025, 1:06:23 PM
URL https://insta-market.de/robots.txt
Domain IPs 104.21.3.180, 172.67.153.152, 2606:4700:3032::6815:3b4, 2606:4700:3036::ac43:9998
Response IP 104.21.3.180
Found Yes
Hash 3e1849f2654eae999679e1b09a08c29abb79d91d2e472ffcb4ac891160c48eac
SimHash 183d494be6b0

Groups

*

Rule Path
Disallow /?k=*

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /cgi-bin/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-signup.php
Disallow /readme.html
Disallow /license.txt

googlebot

Rule Path
Allow /*.js
Allow /*.css

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://insta-market.de/product_cat-sitemap.xml
sitemap https://insta-market.de/product-sitemap.xml
sitemap https://insta-market.de/sitemap_index.xml

Comments

  • Allow all bots to crawl the site
  • Allow Googlebot to crawl CSS and JS files
  • Block specific bots to save crawl budget
  • Sitemap locations