Log In Sign Up

boekenwereld.com
robots.txt

Robots Exclusion Standard data for boekenwereld.com

Resource Scan

Scan Details

Site Domain boekenwereld.com
Base Domain boekenwereld.com
Scan Status Ok
Last Scan2024-10-31T05:03:17+00:00
Next Scan 2024-11-30T05:03:17+00:00

Last Scan

Scanned2024-10-31T05:03:17+00:00
URL https://boekenwereld.com/robots.txt
Redirect https://www.boekenwereld.com/robots.txt
Redirect Domain www.boekenwereld.com
Redirect Base boekenwereld.com
Domain IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Redirect IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Response IP 199.232.46.132
Found Yes
Hash 26ce3cc253c3745f5080d0148a63fb0d90cb0670d761f72d1c701f8c4dbfdf47
SimHash 797fdb71cd11

Groups

*

Rule Path
Disallow /*/q/*_ASC*
Disallow /*/q/*_DESC*

Other Records

Field Value
crawl-delay 25

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

speedyspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.boekenwereld.com/sitemap.xml
Back to top

Comments

  • Website Sitemap
  • Crawlers Setup
  • Paths used by Magento's sorting or layered navigation
  • ----------------------------------------------------
  • Block bad crawlers
  • ----------------------------------------------------
  • third party search engine bot
  • Web-crawler by the company 'WiseGuys' to gather unstructured webdata
  • Chinese Search Engine
  • French Search Engine
  • Russian Search Engine
Back to top