brodyaga.com
robots.txt

Robots Exclusion Standard data for brodyaga.com

Resource Scan

Scan Details

Site Domain brodyaga.com
Base Domain brodyaga.com
Scan Status Ok
Last Scan2024-09-29T02:23:40+00:00
Next Scan 2024-10-29T02:23:40+00:00

Last Scan

Scanned2024-09-29T02:23:40+00:00
URL https://brodyaga.com/robots.txt
Domain IPs 87.249.43.129
Response IP 87.249.43.129
Found Yes
Hash a7bcd566e46f06c293772c4398d75cba1fbbdbb8cda100b6e2f931a86540069f
SimHash f87d5f4a8e92

Groups

*

Rule Path
Disallow /pages/thumbs/
Disallow /pages/bigthumbs/
Disallow /pages/adv/
Disallow /pages/maps/
Disallow /pages/stories/
Disallow /pages/Images/
Disallow /pages/include/
Disallow /pages/admin/
Disallow /pages/un/
Disallow /themes/
Disallow /images/
Disallow /images2002/
Disallow /images2003/
Disallow /images2004/
Disallow /thumbs/
Disallow /thumbs2004/
Disallow /thumbs2003/
Disallow /thumbs2002/

baiduspider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.brodyaga.com/sitemap.xml

Warnings

  • 6 invalid lines.