Log In Sign Up

woodweb.com
robots.txt

Robots Exclusion Standard data for woodweb.com

Resource Scan

Scan Details

Site Domain woodweb.com
Base Domain woodweb.com
Scan Status Ok
Last Scan2024-09-29T18:26:49+00:00
Next Scan 2024-10-06T18:26:49+00:00

Last Scan

Scanned2024-09-29T18:26:49+00:00
URL https://woodweb.com/robots.txt
Domain IPs 67.227.157.78
Response IP 67.227.157.78
Found Yes
Hash 89076b8ab23ee5d9e81d786dfdb88cb9195ac3f4d197ced15843110ee2ce9e8e
SimHash 681ef821f0c0

Groups

*

Rule Path
Disallow /test/
Disallow /terms/
Disallow /web_log/
Disallow /__jr/
Disallow /cgi-bin/__jr/
Disallow /SDD/
Disallow /cgi-bin/un.pl
Disallow /cgi-bin/forums/cancel_notify.pl

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

ocelli

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

aghaven/nutch-1.2

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

typhoeus

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

domain re-animator bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

omgili/0.5 +http://omgili.com

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15
Back to top

Comments

  • User-agent: bingbot
  • Crawl-delay: 30
  • JR - added to slow down aggressive robot
  • JR - added to slow down aggressive robot 04/20/2015
  • JR - added to slow down aggressive robot 05/01/2015
Back to top

Warnings

  • 2 invalid lines.
Back to top