mieuxenseigner.be
robots.txt

Robots Exclusion Standard data for mieuxenseigner.be

Resource Scan

Scan Details

Site Domain mieuxenseigner.be
Base Domain mieuxenseigner.be
Scan Status Ok
Last Scan2024-09-21T07:14:26+00:00
Next Scan 2024-09-28T07:14:26+00:00

Last Scan

Scanned2024-09-21T07:14:26+00:00
URL https://mieuxenseigner.be/robots.txt
Redirect https://www.mieuxenseigner.be/robots.txt
Redirect Domain www.mieuxenseigner.be
Redirect Base mieuxenseigner.be
Domain IPs 192.252.157.152
Redirect IPs 192.252.157.152
Response IP 192.252.157.152
Found Yes
Hash 9cd1beecde1c08dc595bc3da7efcfe6948913585cdf27871fb2d9ca3b8ddd407
SimHash 2c34bd81d7dd

Groups

*

Rule Path
Disallow /*sort%3D
Disallow /*order%3D
Disallow /*limit%3D

*

Rule Path
Disallow /cgi-bin/*
Disallow /image/*
Disallow /javascript/*
Disallow /plugs/*
Disallow /sendy/*
Disallow /var/*
Disallow /etc/*

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

googlebot-video

Rule Path
Disallow

bingbot

Rule Path
Disallow

bingpreview
msnbot

Rule Path
Disallow

msnbot-media

Rule Path
Disallow

slurp

Rule Path
Disallow

facebot

Rule Path
Disallow

pinterest

Rule Path
Disallow

twitterbot

Rule Path
Disallow

orangebot

Rule Path
Disallow

dazoobot

Rule Path
Disallow

qwantify

Rule Path
Disallow

euripbot

Rule Path
Disallow

findxbot

Rule Path
Disallow

istellabot

Rule Path
Disallow

plukkie

Rule Path
Disallow

cliqzbot

Rule Path
Disallow

deusu

Rule Path
Disallow

applebot

Rule Path
Disallow

lycos

Rule Path
Disallow

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

orangebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

Comments

  • Blocks common URL parameters
  • Test directories
  • allowed bots
  • dazoo.fr french search engine
  • qwant.com french search engine
  • eurip.com european search engine
  • findx.com european search engine
  • istella.it italian search engine
  • botje.nl dutch search engine
  • cliqz.com german in-product search engine
  • deusu.de german search engine
  • 120 is the delay in seconds between page requests
  • disallow bad/aggressive bots - also see .htaccess