lexology.com
robots.txt

Robots Exclusion Standard data for lexology.com

Resource Scan

Scan Details

Site Domain lexology.com
Base Domain lexology.com
Scan Status Ok
Last Scan2025-02-22T22:40:29+00:00
Next Scan 2025-03-24T22:40:29+00:00

Last Scan

Scanned2025-02-22T22:40:29+00:00
URL https://lexology.com/robots.txt
Redirect https://www.lexology.com/robots.txt
Redirect Domain www.lexology.com
Redirect Base lexology.com
Domain IPs 104.18.12.102, 104.18.13.102, 2606:4700::6812:c66, 2606:4700::6812:d66
Redirect IPs 104.18.12.102, 104.18.13.102, 2606:4700::6812:c66, 2606:4700::6812:d66
Response IP 104.18.13.102
Found Yes
Hash a6341160f8e99d6ca8d265ed839c4ba32b41940202ad084f97056ce0f85feae5
SimHash 640c1a40e5a2

Groups

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /Admin
Disallow /admin
Disallow /Account
Disallow /account
Disallow /Library/Document.ashx
Disallow /library/Document.ashx
Disallow /Library/document.ashx
Disallow /library/document.ashx
Disallow /Library/ReadItLaterPopup.aspx
Disallow /library/ReadItLaterPopup.aspx
Disallow /r.ashx
Disallow /R.ashx
Disallow /r.ashx?i=*
Disallow /R.ashx?i=*
Disallow /r.ashx?l=*
Disallow /R.ashx?l=*
Disallow /%28X%281
Disallow /%28x%281
Disallow /library/forward.aspx
Disallow /library/account/
Disallow /library/firms/
Disallow /library/images/
Disallow /emailv_v2
Disallow /Emailv_v2
Disallow /newsfeedemail
Disallow /olddefault.aspx
Disallow /OldDefault.aspx
Disallow /oldDefault.aspx
Disallow /Olddefault.aspx
Disallow /Track
Disallow /track
Disallow /r/
Disallow /R/
Disallow /email/r/
Disallow /Email/R/
Disallow /email/R/
Disallow /Email/r/
Allow /Account/Register.aspx
Allow /Account/register.aspx
Allow /account/Register.aspx
Allow /account/register.aspx
Allow /account/rsscontrolpanel.aspx

Other Records

Field Value
sitemap https://www.lexology.com/sitemap.xml