plein.nl
robots.txt

Robots Exclusion Standard data for plein.nl

Resource Scan

Scan Details

Site Domain plein.nl
Base Domain plein.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-11T05:38:43+00:00
Next Scan 2024-07-11T05:38:43+00:00

Last Successful Scan

Scanned2024-05-06T03:10:05+00:00
URL https://plein.nl/robots.txt
Redirect https://www.plein.nl/robots.txt
Redirect Domain www.plein.nl
Redirect Base plein.nl
Domain IPs 104.22.18.174, 104.22.19.174, 172.67.9.168, 2606:4700:10::6816:12ae, 2606:4700:10::6816:13ae, 2606:4700:10::ac43:9a8
Redirect IPs 104.22.18.174, 104.22.19.174, 172.67.9.168, 2606:4700:10::6816:12ae, 2606:4700:10::6816:13ae, 2606:4700:10::ac43:9a8
Response IP 172.67.9.168
Found Yes
Hash 7bad3dbe23449adda3e1835016bca42e92af3ad69199ef4ca331157a684dd6fd
SimHash 7291d951c81a

Groups

*

Rule Path
Disallow /zoeken
Disallow /winkelmand/
Disallow /login
Disallow /cookies/
Disallow /review/
Disallow /reviews/
Disallow /register
Disallow /resetting/request
Disallow /checkout/
Disallow /profile/

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.plein.nl/sitemaps/plein-nl.xml

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • For specific bots