madensektor.com
robots.txt

Robots Exclusion Standard data for madensektor.com

Resource Scan

Scan Details

Site Domain madensektor.com
Base Domain madensektor.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-05-01T06:04:06+00:00
Next Scan 2024-07-30T06:04:06+00:00

Last Successful Scan

Scanned2022-12-12T09:23:58+00:00
URL http://madensektor.com/robots.txt
Domain IPs 38.40.149.230
Response IP 38.40.149.230
Found Yes
Hash 10d3f7f671094d585023d560256e6e6de83dfd2c0df9e97a592fd3e14cbca37a
SimHash 3ad0a84258ea

Groups

lycos_spider_

Rule Path
Disallow /

fast-webcrawler

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

pangusospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

youdaofeedfetcher

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

wbsrch

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

obot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

gluten free crawler

Rule Path
Disallow /

gimmeusabot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

*

Rule Path
Disallow /*.js$
Disallow /*.css$
Disallow /wp-content/
Disallow /wp-includes/

Warnings

  • 2 invalid lines.