bocconilegalpapers.org
robots.txt

Robots Exclusion Standard data for bocconilegalpapers.org

Resource Scan

Scan Details

Site Domain bocconilegalpapers.org
Base Domain bocconilegalpapers.org
Scan Status Ok
Last Scan2024-10-21T16:25:21+00:00
Next Scan 2024-11-20T16:25:21+00:00

Last Scan

Scanned2024-10-21T16:25:21+00:00
URL https://bocconilegalpapers.org/robots.txt
Domain IPs 104.21.44.172, 172.67.201.154, 2606:4700:3033::ac43:c99a, 2606:4700:3035::6815:2cac
Response IP 104.21.44.172
Found Yes
Hash f18272aca0ce062cbc1553b1148c4d8cbb68d7700fdd8b8acc86c21a220543a6
SimHash 6418d1f0c7b2

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /wp-admin/
Disallow /?
Disallow *?s=
Disallow *%26s%3D
Disallow /search
Disallow /author/
Disallow */embed
Disallow */page/
Disallow */xmlrpc.php
Disallow *utm*%3D
Disallow *openstat%3D

ccbot

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

ccbot/2.0 (http://commoncrawl.org/faq/)

Rule Path
Disallow /

wikido

Rule Path
Disallow /

fr_crawler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bitvorebot

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

kraken

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

synthesio

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

brandonbot

Rule Path
Disallow /

germcrawler

Rule Path
Disallow /

sogou

Rule Path
Disallow /

exabot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

admantx

Rule Path
Disallow /
Allow /

Other Records

Field Value
sitemap https://bocconilegalpapers.org/sitemap_index.xml
sitemap https://bocconilegalpapers.org/sitemap_index.xml

Warnings

  • `host` is not a known field.