lawyers.com
robots.txt

Robots Exclusion Standard data for lawyers.com

Resource Scan

Scan Details

Site Domain lawyers.com
Base Domain lawyers.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-01T19:10:58+00:00
Next Scan 2024-10-31T19:10:58+00:00

Last Successful Scan

Scanned2024-06-05T17:44:43+00:00
URL https://lawyers.com/robots.txt
Redirect https://www.lawyers.com/robots.txt
Redirect Domain www.lawyers.com
Redirect Base lawyers.com
Domain IPs 98.158.193.91
Redirect IPs 104.18.35.247, 172.64.152.9
Response IP 172.64.152.9
Found Yes
Hash a40d0590a2be174f2fa279263d27fec8cdcf0d83b00e76d26c931111dd2b64be
SimHash 6ecf13a04390

Groups

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

googleother

Rule Path
Disallow /

yandex

Rule Path
Disallow /

iisbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

*

Rule Path
Disallow /*/answers/
Disallow /*/contact/
Disallow /*/contact/quick/send/
Disallow /*/contact/send/
Disallow /*/?event=
Disallow /sitecore/*
Disallow /*?rp=
Disallow /amp/canada/*
Disallow /ajax/getProfileTopAop/*
Disallow /attorney/
Disallow /*?refPageViewId
Disallow /cdn-cgi/
Disallow /path2/
Disallow /*/update-profile/
Disallow /top-lawyers/
Disallow /*/?filter=
Disallow /*/?review=
Disallow /webstats/

Other Records

Field Value
sitemap https://www.lawyers.com/law_firm_serps.xml
sitemap https://www.lawyers.com/browse_path.xml
sitemap https://www.lawyers.com/attorney_profiles.xml
sitemap https://www.lawyers.com/law_firm_profiles.xml
sitemap https://www.lawyers.com/sitemap_new_profiles.xml