csicop.org
robots.txt

Robots Exclusion Standard data for csicop.org

Resource Scan

Scan Details

Site Domain csicop.org
Base Domain csicop.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-26T20:56:39+00:00
Next Scan 2025-11-25T20:56:39+00:00

Last Successful Scan

Scanned2025-09-04T12:37:46+00:00
URL https://csicop.org/robots.txt
Redirect https://skepticalinquirer.org/robots.txt
Redirect Domain skepticalinquirer.org
Redirect Base skepticalinquirer.org
Domain IPs 67.227.143.240
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.96.1
Found Yes
Hash 1da869a0fb9d1a4015897e1da08566b7c2df83d484eb9d706d82aa661e8e4d14
SimHash 6ab65a32c2a3

Groups

mauibot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /

applebot

Rule Path
Allow /

baiduspider

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

facebot

Rule Path
Allow /

googlebot

Rule Path
Allow /
Allow /wp-content/uploads/
Disallow /activity/
Disallow /?acpage=
Disallow /s/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /*/feed/

msnbot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

slurp

Rule Path
Allow /

teoma

Rule Path
Disallow /

twitterbot

Rule Path
Allow /

yandex

Rule Path
Disallow /

yeti

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /

*

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5