dtcc.edu
robots.txt

Robots Exclusion Standard data for dtcc.edu

Resource Scan

Scan Details

Site Domain dtcc.edu
Base Domain dtcc.edu
Scan Status Ok
Last Scan2024-06-28T12:55:57+00:00
Next Scan 2024-07-28T12:55:57+00:00

Last Scan

Scanned2024-06-28T12:55:57+00:00
URL https://dtcc.edu/robots.txt
Redirect https://www.dtcc.edu/robots.txt
Redirect Domain www.dtcc.edu
Redirect Base dtcc.edu
Domain IPs 75.2.28.147, 99.83.189.201
Redirect IPs 3.212.237.234, 34.196.114.147
Response IP 3.212.237.234
Found Yes
Hash b87802e199b217ffd75802ac4e5d82b32dfbd30ba4548d85d6871dc3386f66e8
SimHash 6115d45167dd

Groups

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

googlebot-image

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

archive.org_bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

ia_archiver

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

cludo

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

siteimprovebot-crawler

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.dtcc.edu/sitemap.xml /* It needs to point to the Google sitemap as per the build*/