mccombs.utexas.edu
robots.txt

Robots Exclusion Standard data for mccombs.utexas.edu

Resource Scan

Scan Details

Site Domain mccombs.utexas.edu
Base Domain utexas.edu
Scan Status Ok
Last Scan2024-05-30T20:07:15+00:00
Next Scan 2024-06-29T20:07:15+00:00

Last Scan

Scanned2024-05-30T20:07:15+00:00
URL https://mccombs.utexas.edu/robots.txt
Domain IPs 13.248.167.164, 76.223.50.213
Response IP 13.248.167.164
Found Yes
Hash 472035b82ee0cfc9acffa2b7930ac28c65dc749316f560fa5c73927415273a2e
SimHash 6518dc13645f

Groups

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

googlebot-image

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

funnelback

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.mccombs.utexas.edu/sitemap-en.xml