Log In Sign Up

venauto.nl
robots.txt

Robots Exclusion Standard data for venauto.nl

Resource Scan

Scan Details

Site Domain venauto.nl
Base Domain venauto.nl
Scan Status Ok
Last Scan2024-09-29T22:07:59+00:00
Next Scan 2024-10-29T22:07:59+00:00

Last Scan

Scanned2024-09-29T22:07:59+00:00
URL https://venauto.nl/robots.txt
Redirect https://www.venauto.nl/robots.txt
Redirect Domain www.venauto.nl
Redirect Base venauto.nl
Domain IPs 104.26.8.120, 104.26.9.120, 172.67.73.141, 2606:4700:20::681a:878, 2606:4700:20::681a:978, 2606:4700:20::ac43:498d
Redirect IPs 104.26.8.120, 104.26.9.120, 172.67.73.141, 2606:4700:20::681a:878, 2606:4700:20::681a:978, 2606:4700:20::ac43:498d
Response IP 172.67.73.141
Found Yes
Hash 6b7424e62533ba09d858e59c619adb475c0ca52e9cc0d426843db08d5e31336e
SimHash ef0c5805c2b9

Groups

slurp

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

*

Rule Path
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /magento/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /yd/
Disallow /feed/
Disallow /import/
Disallow /magmi/
Disallow /mt940/
Disallow /promasy/
Disallow /am_feed/
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /control/
Disallow /contacts/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /landingpages/
Disallow /checkout/
Disallow /*?p=*&
Disallow /*?SID=
Disallow /*?limit=
Disallow /*?price=
Disallow /*ajax%3D
Disallow /*dir%3D
Disallow /*order%3D
Disallow /*p%3D1$
Disallow *PHPSESSID
Disallow /voorwaarden/s
Disallow /disclaimer/
Disallow /catalog/seo_sitemap/product/

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap http://www.venauto.nl/sitemap/nl/sitemapindex.xml
Back to top

Comments

  • Robots.txt
  • Disable Yahoo! crawler: https://help.yahoo.com/kb/search/SLN22600.html
  • https://es.semrush.com/bot/
  • http://www.twenga.com/bot.html
  • User-agent: TwengaBot-2.0
  • Disallow: /
  • http://mj12bot.com/index.html
  • https://ahrefs.com/robot
  • https://ahrefs.com/robot
  • https://moz.com/researchtools/ose/dotbot
  • http://webmeup-crawler.com/
  • Other crawlers setup
  • Directories Magento
  • Paths (clean URLs) Magento
  • Paths (no clean URLs) Magento
  • Disallow: /*.php$
  • Niet relevante pagina's
  • Sitemaps
Back to top