niagara.craigslist.org
robots.txt

Robots Exclusion Standard data for niagara.craigslist.org

Resource Scan

Scan Details

Site Domain niagara.craigslist.org
Base Domain craigslist.org
Scan Status Ok
Last Scan2024-10-05T05:31:02+00:00
Next Scan 2024-10-19T05:31:02+00:00

Last Scan

Scanned2024-10-05T05:31:02+00:00
URL https://niagara.craigslist.org/robots.txt
Domain IPs 208.82.238.242
Response IP 208.82.238.2
Found Yes
Hash 267754ee0dc054a36e05d42bc2c7d7e7c43821f86b2e17266384f0abf46dfc32
SimHash d1185911a4e7

Groups

*

Rule Path
Disallow /reply
Disallow /fb/
Disallow /suggest
Disallow /flag
Disallow /mf
Disallow /mailflag
Disallow /eaf

amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
friendlycrawler
google-extended
gptbot
imagesiftbot
img2dataset
meta-externalagent
oai-searchbot
omgili
omgilibot
perplexitybot
timpibot
velenpublicwebcrawler
youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://niagara.craigslist.org/sitemap/index.xml
sitemap https://niagara.craigslist.org/sitemap/index-by-date-cat.xml