vivara.nl
robots.txt

Robots Exclusion Standard data for vivara.nl

Resource Scan

Scan Details

Site Domain vivara.nl
Base Domain vivara.nl
Scan Status Ok
Last Scan2024-06-24T23:58:20+00:00
Next Scan 2024-07-24T23:58:20+00:00

Last Scan

Scanned2024-06-24T23:58:20+00:00
URL https://vivara.nl/robots.txt
Redirect https://www.vivara.nl/robots.txt
Redirect Domain www.vivara.nl
Redirect Base vivara.nl
Domain IPs 3.78.162.126
Redirect IPs 3.126.115.145
Response IP 3.126.115.145
Found Yes
Hash ebb6c15fdb7c9a749e74128dd41bd1c3ef903c7b64ce0a84fba2ee38f031af4d
SimHash 533551ca4bd0

Groups

*

Rule Path
Disallow /customer/
Disallow /wishlist/
Disallow /checkout/
Disallow /catalog/
Disallow /catalogsearch/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /onestepcheckout/
Disallow /phpserver/
Disallow /pkginfo/
Disallow /report/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /var/
Allow /*?p=
Disallow /*?*
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /package.json
Disallow /RELEASE_NOTES.txt
Disallow /*.sample
Disallow /*.git
Disallow /*.cvs
Disallow /*.zip$
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$

httrack

Rule Path
Disallow /

pathdefender

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

qihoobot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

cityreview

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

iisbot

Rule Path
Disallow /

screaming frog

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

custo

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

newspointbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

abonti

Rule Path
Disallow /

buibui-bot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

zyklop

Rule Path
Disallow /

bubing

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

jsoup

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

domain re-animator

Rule Path
Disallow /

admantx

Rule Path
Disallow /

qcrawl

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

onpagebot

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

Comments

  • Directories
  • Parameters
  • Files
  • Specific Crawlers