samueli.ucla.edu
robots.txt

Robots Exclusion Standard data for samueli.ucla.edu

Resource Scan

Scan Details

Site Domain samueli.ucla.edu
Base Domain ucla.edu
Scan Status Ok
Last Scan2025-03-03T23:34:04+00:00
Next Scan 2025-04-02T23:34:04+00:00

Last Scan

Scanned2025-03-03T23:34:04+00:00
URL https://samueli.ucla.edu/robots.txt
Domain IPs 164.67.100.181
Response IP 164.67.100.181
Found Yes
Hash f208bb3a407cd29c0ad115b0e5eb8943405caac509693fe08f362472cf06ad85
SimHash 295d5e44ec9b

Groups

*

Rule Path
Disallow /content-*
Disallow /wp-admin/*
Disallow /author/*
Disallow /category/uncategorized/*

Other Records

Field Value
crawl-delay 5

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

sitecheck

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

Other Records

Field Value
sitemap https://samueli.ucla.edu/sitemap.xml