shohanur-rahman.xyz
robots.txt

Robots Exclusion Standard data for shohanur-rahman.xyz

Resource Scan

Scan Details

Site Domain shohanur-rahman.xyz
Base Domain shohanur-rahman.xyz
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-09-12T07:45:33+00:00
Next Scan 2025-12-11T07:45:33+00:00

Last Successful Scan

Scanned2024-01-28T22:24:18+00:00
URL http://shohanur-rahman.xyz/robots.txt
Domain IPs 38.14.39.168
Response IP 38.14.39.168
Found Yes
Hash c2638717a929cf78ac4cb4b536c12da3a64e9eb858f1f72f2b34329165eab214
SimHash 3ad4a04248ca

Groups

lycos_spider_

Rule Path
Disallow /

fast-webcrawler

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

pangusospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

youdaofeedfetcher

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

wbsrch

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

obot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

gluten free crawler

Rule Path
Disallow /

gimmeusabot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow /*.js$
Disallow /*.css$
Disallow /wp-content/
Disallow /wp-includes/

Warnings

  • 2 invalid lines.