Log In Sign Up

boeknet.nl
robots.txt

Robots Exclusion Standard data for boeknet.nl

Resource Scan

Scan Details

Site Domain boeknet.nl
Base Domain boeknet.nl
Scan Status Ok
Last Scan2024-09-10T19:23:20+00:00
Next Scan 2024-10-10T19:23:20+00:00

Last Scan

Scanned2024-09-10T19:23:20+00:00
URL https://boeknet.nl/robots.txt
Redirect https://www.boeknet.nl/robots.txt
Redirect Domain www.boeknet.nl
Redirect Base boeknet.nl
Domain IPs 77.174.163.233
Redirect IPs 77.174.163.233
Response IP 77.174.163.233
Found Yes
Hash 402a19c8ecea8ac1786943d1085218dbba3a96cabe5242a74c05ea1a7c29ad5d
SimHash c91c9847c291

Groups

googlebot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ezooms

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

megaindex.com

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

megaindex.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

megaindex.ru/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /admin/
Disallow /veelgestelde-vragen/
Disallow /disclaimer/
Disallow /search/
Back to top

Other Records

Field Value
sitemap http://www.boeknet.nl/sitemap.xml
Back to top

Comments

  • User-agent: AhrefsBot
  • Crawl-delay: 20
Back to top