listcompanies.in
robots.txt

Robots Exclusion Standard data for listcompanies.in

Resource Scan

Scan Details

Site Domain listcompanies.in
Base Domain listcompanies.in
Scan Status Ok
Last Scan2024-10-07T11:27:58+00:00
Next Scan 2024-10-14T11:27:58+00:00

Last Scan

Scanned2024-10-07T11:27:58+00:00
URL http://listcompanies.in/robots.txt
Response IP 223.25.237.163
Found Yes
Hash 29abc646738679d9edb557205cb6578c7226e84c9911f02d37b0461cf1ddc8e4
SimHash 201fc8306c9b

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /admin

baiduspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

microsoft.url

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /