americanbar.org
robots.txt

Robots Exclusion Standard data for americanbar.org

Resource Scan

Scan Details

Site Domain americanbar.org
Base Domain americanbar.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-26T19:13:55+00:00
Next Scan 2024-09-24T19:13:55+00:00

Last Successful Scan

Scanned2023-08-08T12:47:29+00:00
URL https://americanbar.org/robots.txt
Redirect https://www.americanbar.org/robots.txt
Redirect Domain www.americanbar.org
Redirect Base americanbar.org
Domain IPs 45.60.122.186, 45.60.132.186
Redirect IPs 45.60.126.186
Response IP 45.60.126.186
Found Yes
Hash b4840e935b8a413f72485b1b4b48681cf3a948be66352308d9793d702ef46185
SimHash 7002865a4727

Groups

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

twitterbot

Rule Path
Allow /content/dam/

facebot

Rule Path
Allow /content/dam/

*

Rule Path
Allow /content/dam/aba-cms-dotorg/products/
Allow /content/dam/aba/root
Allow /content/dam/aba/publications
Allow /content/dam/aba/publishing
Disallow /tools/
Disallow /errors/
Disallow /sandbox/
Disallow /test/
Disallow /directories/
Disallow /auth/
Disallow /claim/
Disallow /my-aba/
Disallow /join/
Disallow /checkout/
Disallow /cart/
Disallow /content/dam/
Disallow /content/dam/aba/administrative/aba/cyberalert/
Disallow /content/dam/aba/administrative/cyberalert/
Disallow /content/dam/aba/administrative/aba/ransomware.pdf
Disallow /content/dam/aba/administrative/aba/ransomware.authcheckdam.pdf
Disallow /events-cle/mtg/inviteonly/
Disallow /profile*

Other Records

Field Value
sitemap https://www.americanbar.org/content/dam/aba/root/sitemap.gz
sitemap https://www.americanbar.org/content/dam/aba/root/sitemap2.gz