avatarpress.com
robots.txt

Robots Exclusion Standard data for avatarpress.com

Resource Scan

Scan Details

Site Domain avatarpress.com
Base Domain avatarpress.com
Scan Status Ok
Last Scan2024-05-22T16:28:40+00:00
Next Scan 2024-06-21T16:28:40+00:00

Last Scan

Scanned2024-05-22T16:28:40+00:00
URL http://www.avatarpress.com/robots.txt
Domain IPs 34.195.158.81
Response IP 34.195.158.81
Found Yes
Hash ec9601e1cf4f10b3dfb6e90328335461ff4bfb82ff220c391cf96166f6ddd470
SimHash 629ad0c08193

Groups

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex/1.01.001

Rule Path
Disallow /

yandexbot/3.0-mirrordetector

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yandexsomething/1.

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

discobot

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /wp-content/plugins/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/themes/
Disallow /?s=*

Warnings

  • 4 invalid lines.