/.well-known/

Log In Sign Up

justindianpornx.com
robots.txt

Robots Exclusion Standard data for justindianpornx.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	justindianpornx.com
Base Domain	justindianpornx.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-08-30T03:39:07+00:00
Next Scan	2024-10-29T03:39:07+00:00

Last Successful Scan

Scanned	2024-07-02T02:59:15+00:00
URL	https://justindianpornx.com/robots.txt
Domain IPs	104.21.4.157, 172.67.132.59, 2606:4700:3034::6815:49d, 2606:4700:3037::ac43:843b
Response IP	172.67.132.59
Found	Yes
Hash	23d046f05bbe23b4363bb3433b6d84fc38f0a396d8db6d113a54b214171397d9
SimHash	505dd1c0ecb9

Groups

*

Rule

Path

Disallow

/*.php*

Disallow

/?s=*

blexbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

vagabondo

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

archive.org_bot

Rule

Path

Disallow

/

special_archiver

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

special_archiver

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

queryseekerspider

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

baiduspider

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

30

msnbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

7

bingbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

7

Back to top

Warnings

2 invalid lines.

Back to top