Log In Sign Up

nsesoftware.nl
robots.txt

Robots Exclusion Standard data for nsesoftware.nl

Resource Scan

Scan Details

Site Domain nsesoftware.nl
Base Domain nsesoftware.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-28T17:46:33+00:00
Next Scan 2025-09-26T17:46:33+00:00

Last Successful Scan

Scanned2023-11-12T04:05:18+00:00
URL https://nsesoftware.nl/robots.txt
Domain IPs 2a01:238:20a:202:1163::, 81.169.145.163
Response IP 81.169.145.163
Found Yes
Hash bc46ba8d7648d503056ccb4d454e5f88a38f7b37e4085db896b4db625eda7d69
SimHash 0714d2e3e4d1

Groups

duckduckbot

Rule Path
Allow /

yandexbot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msnbot/2.1

Rule Path
Disallow /

msnbot/2.0b

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

amazon-kendra

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /
Back to top

Comments

  • User-agents:
  • *, Googlebot, Googlebot-Image, Mediapartners-Google, Googlebot-news,
  • msnbot, AspiegelBot (Huawei), PetalBot (?) (Huawei)
  • Zie ook: https://developers.google.com/search/docs/advanced/robots/create-robots-txt
  • https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers
  • https://simtechdev.com/blog/good-and-bad-bots-to-control-to-save-server-resources-and-improve-performance/
  • User-agent: *
  • Disallow: /
  • Crawl-Delay: 10 (seconden)
  • 20230609
  • 20220503:
Back to top