Log In Sign Up

nytcrossword.org
robots.txt

Robots Exclusion Standard data for nytcrossword.org

Resource Scan

Scan Details

Site Domain nytcrossword.org
Base Domain nytcrossword.org
Scan Status Ok
Last Scan2024-11-11T16:54:28+00:00
Next Scan 2024-11-18T16:54:28+00:00

Last Scan

Scanned2024-11-11T16:54:28+00:00
URL https://nytcrossword.org/robots.txt
Domain IPs 2a01:4ff:f0:e30b::1, 5.161.83.186
Response IP 5.161.83.186
Found Yes
Hash 1608941824347a00c2ffee014182a7998a44dff05feffe56d8387ed7f1763cb2
SimHash 70105940c135

Groups

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

*

Rule Path
Disallow /*/1000$
Back to top

Comments

  • AI and LLM Crawling
Back to top