Log In Sign Up

crimewatchdaily.com
robots.txt

Robots Exclusion Standard data for crimewatchdaily.com

Resource Scan

Scan Details

Site Domain crimewatchdaily.com
Base Domain crimewatchdaily.com
Scan Status Ok
Last Scan2024-09-19T15:51:51+00:00
Next Scan 2024-09-26T15:51:51+00:00

Last Scan

Scanned2024-09-19T15:51:51+00:00
URL https://crimewatchdaily.com/robots.txt
Redirect https://truecrimenews.com/robots.txt
Redirect Domain truecrimenews.com
Redirect Base truecrimenews.com
Domain IPs 13.33.88.127, 13.33.88.15, 13.33.88.73, 13.33.88.80, 2600:9000:223b:3000:f:8ded:5f80:93a1, 2600:9000:223b:aa00:f:8ded:5f80:93a1, 2600:9000:223b:ac00:f:8ded:5f80:93a1, 2600:9000:223b:c00:f:8ded:5f80:93a1, 2600:9000:223b:d200:f:8ded:5f80:93a1, 2600:9000:223b:d800:f:8ded:5f80:93a1, 2600:9000:223b:ec00:f:8ded:5f80:93a1, 2600:9000:223b:f800:f:8ded:5f80:93a1
Redirect IPs 13.33.30.110, 13.33.30.118, 13.33.30.123, 13.33.30.6, 2600:9000:229f:3200:d:d308:75c0:93a1, 2600:9000:229f:4800:d:d308:75c0:93a1, 2600:9000:229f:4a00:d:d308:75c0:93a1, 2600:9000:229f:4c00:d:d308:75c0:93a1, 2600:9000:229f:5600:d:d308:75c0:93a1, 2600:9000:229f:9800:d:d308:75c0:93a1, 2600:9000:229f:dc00:d:d308:75c0:93a1, 2600:9000:229f:fa00:d:d308:75c0:93a1
Response IP 13.33.30.118
Found Yes
Hash f077f93b8e0ba412c288abdcec827ce4a895374fe09a85953336f6a1988ce715
SimHash 6d3c5410f6fb

Groups

googlebot-news

Rule Path
Disallow /photos
Disallow /videos

proximic

Rule Path
Disallow

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /_/
Disallow */print
Disallow /search
Back to top

Other Records

Field Value
sitemap https://truecrimenews.com/sitemaps/article/index.xml
sitemap https://truecrimenews.com/sitemaps/gallery/index.xml
sitemap https://truecrimenews.com/sitemaps/page/index.xml
sitemap https://truecrimenews.com/sitemaps/video/index.xml
sitemap https://truecrimenews.com/sitemaps/news.xml
Back to top