Log In Sign Up

cdnc.ucr.edu
robots.txt

Robots Exclusion Standard data for cdnc.ucr.edu

Resource Scan

Scan Details

Site Domain cdnc.ucr.edu
Base Domain ucr.edu
Scan Status Ok
Last Scan2025-03-03T12:31:19+00:00
Next Scan 2025-04-02T12:31:19+00:00

Last Scan

Scanned2025-03-03T12:31:19+00:00
URL https://cdnc.ucr.edu/robots.txt
Domain IPs 138.23.243.73
Response IP 138.23.243.73
Found Yes
Hash 62a1bc5976704d21404cbcd7645004f54d11c4d610ca8759d6d3f208eb9db09e
SimHash 6e7c9d71e775

Groups

*

Rule Path
Disallow /

applebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

archive.org_bot

Rule Path Comment
Disallow /cgi-bin/imageserver.pl Block imageserver requests
Disallow /?a=a Block authentication
Disallow /?a=cb Block content blocking
Disallow /?a=gs Block graphserver requests
Disallow /?a=is Block imageserver requests
Disallow /?a=pme Block publication metadata editor
Disallow /?a=q Block queries
Disallow /?a=tc Block text correction
Disallow /?a=ua Block user account
Disallow /?a=ur Block user registration
Disallow /?*&e=* Block all internal links
Disallow /cgi-bin/jewishweekly?a=a Block authentication
Disallow /cgi-bin/jewishweekly?a=cb Block content blocking
Disallow /cgi-bin/jewishweekly?a=gs Block graphserver requests
Disallow /cgi-bin/jewishweekly?a=is Block imageserver requests
Disallow /cgi-bin/jewishweekly?a=pme Block publication metadata editor
Disallow /cgi-bin/jewishweekly?a=q Block queries
Disallow /cgi-bin/jewishweekly?a=tc Block text correction
Disallow /cgi-bin/jewishweekly?a=ua Block user account
Disallow /cgi-bin/jewishweekly?a=ur Block user registration
Disallow /cgi-bin/jewishweekly?*&e=* Block all internal links
Disallow */iiif/ Block all IIIF requests

Other Records

Field Value Comment
crawl-delay 1 Wait one second between requests

bingbot

Rule Path Comment
Disallow /cgi-bin/imageserver.pl Block imageserver requests
Disallow /?a=a Block authentication
Disallow /?a=cb Block content blocking
Disallow /?a=gs Block graphserver requests
Disallow /?a=is Block imageserver requests
Disallow /?a=pme Block publication metadata editor
Disallow /?a=q Block queries
Disallow /?a=tc Block text correction
Disallow /?a=ua Block user account
Disallow /?a=ur Block user registration
Disallow /?*&e=* Block all internal links
Disallow /cgi-bin/jewishweekly?a=a Block authentication
Disallow /cgi-bin/jewishweekly?a=cb Block content blocking
Disallow /cgi-bin/jewishweekly?a=gs Block graphserver requests
Disallow /cgi-bin/jewishweekly?a=is Block imageserver requests
Disallow /cgi-bin/jewishweekly?a=pme Block publication metadata editor
Disallow /cgi-bin/jewishweekly?a=q Block queries
Disallow /cgi-bin/jewishweekly?a=tc Block text correction
Disallow /cgi-bin/jewishweekly?a=ua Block user account
Disallow /cgi-bin/jewishweekly?a=ur Block user registration
Disallow /cgi-bin/jewishweekly?*&e=* Block all internal links
Disallow */iiif/ Block all IIIF requests

Other Records

Field Value Comment
crawl-delay 1 Wait one second between requests

googlebot

Rule Path Comment
Disallow /cgi-bin/imageserver.pl Block imageserver requests
Disallow /?a=a Block authentication
Disallow /?a=cb Block content blocking
Disallow /?a=gs Block graphserver requests
Disallow /?a=is Block imageserver requests
Disallow /?a=pme Block publication metadata editor
Disallow /?a=q Block queries
Disallow /?a=tc Block text correction
Disallow /?a=ua Block user account
Disallow /?a=ur Block user registration
Disallow /?*&e=* Block all internal links
Disallow /cgi-bin/jewishweekly?a=a Block authentication
Disallow /cgi-bin/jewishweekly?a=cb Block content blocking
Disallow /cgi-bin/jewishweekly?a=gs Block graphserver requests
Disallow /cgi-bin/jewishweekly?a=is Block imageserver requests
Disallow /cgi-bin/jewishweekly?a=pme Block publication metadata editor
Disallow /cgi-bin/jewishweekly?a=q Block queries
Disallow /cgi-bin/jewishweekly?a=tc Block text correction
Disallow /cgi-bin/jewishweekly?a=ua Block user account
Disallow /cgi-bin/jewishweekly?a=ur Block user registration
Disallow /cgi-bin/jewishweekly?*&e=* Block all internal links
Disallow */iiif/ Block all IIIF requests

Other Records

Field Value Comment
crawl-delay 1 Wait one second between requests

yahoo

Rule Path Comment
Disallow /cgi-bin/imageserver.pl Block imageserver requests
Disallow /?a=a Block authentication
Disallow /?a=cb Block content blocking
Disallow /?a=gs Block graphserver requests
Disallow /?a=is Block imageserver requests
Disallow /?a=pme Block publication metadata editor
Disallow /?a=q Block queries
Disallow /?a=tc Block text correction
Disallow /?a=ua Block user account
Disallow /?a=ur Block user registration
Disallow /?*&e=* Block all internal links
Disallow /cgi-bin/jewishweekly?a=a Block authentication
Disallow /cgi-bin/jewishweekly?a=cb Block content blocking
Disallow /cgi-bin/jewishweekly?a=gs Block graphserver requests
Disallow /cgi-bin/jewishweekly?a=is Block imageserver requests
Disallow /cgi-bin/jewishweekly?a=pme Block publication metadata editor
Disallow /cgi-bin/jewishweekly?a=q Block queries
Disallow /cgi-bin/jewishweekly?a=tc Block text correction
Disallow /cgi-bin/jewishweekly?a=ua Block user account
Disallow /cgi-bin/jewishweekly?a=ur Block user registration
Disallow /cgi-bin/jewishweekly?*&e=* Block all internal links
Disallow */iiif/ Block all IIIF requests

Other Records

Field Value Comment
crawl-delay 1 Wait one second between requests
Back to top

Other Records

Field Value
sitemap https://cdnc.ucr.edu/sitemap-cdnc-index.xml
sitemap https://cdnc.ucr.edu/sitemap-jewishweekly-index.xml
sitemap https://cdnc.ucr.edu/sitemap-cdnc-index.xml
sitemap https://cdnc.ucr.edu/sitemap-jewishweekly-index.xml
sitemap https://cdnc.ucr.edu/sitemap-cdnc-index.xml
sitemap https://cdnc.ucr.edu/sitemap-jewishweekly-index.xml
sitemap https://cdnc.ucr.edu/sitemap-cdnc-index.xml
sitemap https://cdnc.ucr.edu/sitemap-jewishweekly-index.xml
Back to top

Comments

  • Veridian robots.txt file
  • =========================================================
  • Block all web crawlers from accessing the site
  • =========================================================
  • Applebot needs to be specifically told to stop; otherwise, it follows Googlebot
  • Explicitly state that it is not for Google’s machine learning purposes
  • =========================================================
  • Only allow the following bots
  • =========================================================
Back to top