/.well-known/

Log In Sign Up

toporn.tv
robots.txt

Robots Exclusion Standard data for toporn.tv

Archived Snapshots

Resource Scan

Scan Details

Site Domain	toporn.tv
Base Domain	toporn.tv
Scan Status	Ok
Last Scan	2024-11-12T06:07:52+00:00
Next Scan	2024-12-12T06:07:52+00:00

Last Scan

Scanned	2024-11-12T06:07:52+00:00
URL	https://toporn.tv/robots.txt
Domain IPs	104.21.55.28, 172.67.144.38, 2606:4700:3033::6815:371c, 2606:4700:3035::ac43:9026
Response IP	172.67.144.38
Found	Yes
Hash	31e94c1734f8536c5bcdd78d6800725b9e425ce7efc45ce20023fc4023037076
SimHash	4f1cc500cab8

Groups

*

Rule

Path

Disallow

/*.json$

Disallow

/*.js$

Disallow

/api/

Disallow

/e/

Disallow

/health/

Disallow

/_page-perf/

Disallow

/_player-perf/

Disallow

/_client-error/

msnbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

bingbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

yandexbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

baiduspider

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

sogou

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

semrushbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

Allow

/$

blexbot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

semrushbot-ba

Rule

Path

Disallow

/

semrushbot-si

Rule

Path

Disallow

/

semrushbot-swa

Rule

Path

Disallow

/

semrushbot-ct

Rule

Path

Disallow

/

semrushbot-bm

Rule

Path

Disallow

/

splitsignalbot

Rule

Path

Disallow

/

amazonbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

Back to top

Other Records

Field

Value

sitemap

https://toporn.tv/sitemap.xml

Back to top

Warnings

2 invalid lines.

Back to top