Log In Sign Up

westnic.net
robots.txt

Robots Exclusion Standard data for westnic.net

Resource Scan

Scan Details

Site Domain westnic.net
Base Domain westnic.net
Scan Status Ok
Last Scan2025-05-06T13:38:59+00:00
Next Scan 2025-06-05T13:38:59+00:00

Last Scan

Scanned2025-05-06T13:38:59+00:00
URL https://westnic.net/robots.txt
Domain IPs 191.101.172.252
Response IP 191.101.172.252
Found Yes
Hash d42b248e35b539a59368caf27ea61d08045a63a90640811eee68e9d5c7038369
SimHash 98511f0045f5

Groups

googlebot

Rule Path
Disallow /cgi-bin

gulliver

Rule Path
Disallow /cgi-bin

slurp

Rule Path
Disallow /cgi-bin

fast

Rule Path
Disallow /cgi-bin

scooter

Rule Path
Disallow /cgi-bin

vscooter

Rule Path
Disallow /cgi-bin

nutch

Rule Path
Disallow /cgi-bin

fast-webcrawler

Rule Path
Disallow /cgi-bin

teoma_agent1

Rule Path
Disallow /cgi-bin

lycos_spider_(t-rex)

Rule Path
Disallow /cgi-bin

msnbot/0.1

Rule Path
Disallow /cgi-bin

architextspider

Rule Path
Disallow /cgi-bin

zyborg

Rule Path
Disallow /cgi-bin

surveybot

Rule Path
Disallow /cgi-bin
Back to top

Comments

  • The crawlers listed below are allowed on the westnic.net.
  • Disallow all other crawlers (future)
Back to top