columbiabarnardhillel.org
robots.txt

Robots Exclusion Standard data for columbiabarnardhillel.org

Resource Scan

Scan Details

Site Domain columbiabarnardhillel.org
Base Domain columbiabarnardhillel.org
Scan Status Ok
Last Scan2024-09-18T03:00:42+00:00
Next Scan 2024-10-18T03:00:42+00:00

Last Scan

Scanned2024-09-18T03:00:42+00:00
URL https://columbiabarnardhillel.org/robots.txt
Domain IPs 104.21.85.222, 172.67.211.193, 2606:4700:3033::6815:55de, 2606:4700:3035::ac43:d3c1
Response IP 172.67.211.193
Found Yes
Hash 57fc7092d4fbd20ea8f2f506593a35e3c163fa8f70ea32c6dd592e6193f770a3
SimHash 4f1edf5056b4

Groups

googlebot
googlebot-image
googlebot-news
googlebot-video
storebot-googlebot
google-inspectiontool
googleother
google-extended
apis-google
adsbot-google-mobile
adsbot-google
mediapartners-google
feedfetcher-google
google-safety
googleproducer

Rule Path
Allow /

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

awariobot
awariosmartbot
aihitbot
barkrowler
baiduspider
baiduspider-render
blexbot
buck
bytespider
infotigerbot
seznambot
sogou
mail.ru_bot
megaindex.ru
mj12bot
petalbot
wellknownbot
yandexbot

Rule Path
Disallow /