breakbulk.news
robots.txt

Robots Exclusion Standard data for breakbulk.news

Resource Scan

Scan Details

Site Domain breakbulk.news
Base Domain breakbulk.news
Scan Status Ok
Last Scan2024-10-06T08:30:37+00:00
Next Scan 2024-10-13T08:30:37+00:00

Last Scan

Scanned2024-10-06T08:30:37+00:00
URL https://breakbulk.news/robots.txt
Domain IPs 185.196.102.111
Response IP 185.196.102.111
Found Yes
Hash a375d9e31605ce5d43d8f4029afbacaf29e5df82ccdea33eac68c9c0504d3cc0
SimHash 4b0f506afe90

Groups

scrapy

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

blp_bbot/0.1

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

lcc

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

Warnings

  • 2 invalid lines.