Log In Sign Up

kerstversiering.nl
robots.txt

Robots Exclusion Standard data for kerstversiering.nl

Resource Scan

Scan Details

Site Domain kerstversiering.nl
Base Domain kerstversiering.nl
Scan Status Ok
Last Scan2024-11-07T09:59:16+00:00
Next Scan 2024-12-07T09:59:16+00:00

Last Scan

Scanned2024-11-07T09:59:16+00:00
URL https://kerstversiering.nl/robots.txt
Redirect https://www.kerstversiering.nl/robots.txt
Redirect Domain www.kerstversiering.nl
Redirect Base kerstversiering.nl
Domain IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Redirect IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Response IP 199.232.46.132
Found Yes
Hash 9912d9ede29fba77f6d77d4b6234bcea468542e75bde05074912632501c9ceec
SimHash 417edb78ed13

Groups

*

Rule Path
Disallow /*/q/*_ASC*
Disallow /*/q/*_DESC*
Disallow /*/q/*
Disallow /search/
Disallow /*/search/

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

speedyspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.kerstversiering.nl/sitemap/sitemap_nl.xml
Back to top

Comments

  • Website Sitemap
  • Crawlers Setup
  • Paths used by Magento's sorting or layered navigation
  • Disallow all filters, search queries, sorting and pagination
  • Disallow the search
  • Disallow with store code
  • ----------------------------------------------------
  • Block bad crawlers
  • ----------------------------------------------------
  • third party search engine bot
  • Web-crawler by the company 'WiseGuys' to gather unstructured webdata
  • Chinese Search Engine
  • French Search Engine
  • Russian Search Engine
  • User-agent: AhrefsBot
  • Disallow: /
Back to top