/.well-known/

Log In Sign Up

justindianxporn.com
robots.txt

Robots Exclusion Standard data for justindianxporn.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	justindianxporn.com
Base Domain	justindianxporn.com
Scan Status	Ok
Last Scan	2024-09-18T10:36:45+00:00
Next Scan	2024-10-18T10:36:45+00:00

Last Scan

Scanned	2024-09-18T10:36:45+00:00
URL	https://justindianxporn.com/robots.txt
Domain IPs	104.21.64.182, 172.67.154.76, 2606:4700:3030::6815:40b6, 2606:4700:3031::ac43:9a4c
Response IP	172.67.154.76
Found	Yes
Hash	23d046f05bbe23b4363bb3433b6d84fc38f0a396d8db6d113a54b214171397d9
SimHash	505dd1c0ecb9

Groups

*

Rule

Path

Disallow

/*.php*

Disallow

/?s=*

blexbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

vagabondo

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

archive.org_bot

Rule

Path

Disallow

/

special_archiver

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

special_archiver

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

queryseekerspider

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

baiduspider

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

30

msnbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

7

bingbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

7

Back to top

Warnings

2 invalid lines.

Back to top