yalebooks.com
robots.txt

Robots Exclusion Standard data for yalebooks.com

Resource Scan

Scan Details

Site Domain yalebooks.com
Base Domain yalebooks.com
Scan Status Ok
Last Scan2025-10-15T21:54:41+00:00
Next Scan 2025-11-14T21:54:41+00:00

Last Scan

Scanned2025-10-15T21:54:41+00:00
URL https://yalebooks.com/robots.txt
Redirect https://yalebooks.yale.edu/robots.txt
Redirect Domain yalebooks.yale.edu
Redirect Base yale.edu
Domain IPs 23.185.0.4
Redirect IPs 23.185.0.4
Response IP 23.185.0.4
Found Yes
Hash 145df938db293e4a27fdee119b67096b56e943d9db1c1a06fdd7b2745f58fae3
SimHash 6805d001c1b7

Groups

*

Rule Path
Disallow /

applebot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

bingbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

discordbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

duckduckbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

facebookexternalhit

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

googlebot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

googlebot-image

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

ia_archiver

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

linkedinbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

msnbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

pinterestbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

slurp

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

teoma

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

twitterbot

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

yandex

Rule Path
Disallow /wp-admin/
Disallow /search*
Allow /
Allow /wp-admin/admin-ajax.php

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://yalebooks.yale.edu/sitemap_index.xml