lawandcrime.com
robots.txt

Robots Exclusion Standard data for lawandcrime.com

Resource Scan

Scan Details

Site Domain lawandcrime.com
Base Domain lawandcrime.com
Scan Status Ok
Last Scan2024-04-24T23:04:18+00:00
Next Scan 2024-05-01T23:04:18+00:00

Last Scan

Scanned2024-04-24T23:04:18+00:00
URL https://lawandcrime.com/robots.txt
Domain IPs 104.22.20.95, 104.22.21.95, 172.67.10.184, 2606:4700:10::6816:145f, 2606:4700:10::6816:155f, 2606:4700:10::ac43:ab8
Response IP 172.67.10.184
Found Yes
Hash 90b46e9988f17a8941646d12462e8c3cbe230a89a4b892b5669bf3b739e6611a
SimHash 4805406281b2

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://lawandcrime.com/sitemap.xml
sitemap https://lawandcrime.com/sitemap.rss