/.well-known/

Log In Sign Up

pornbay.tv
robots.txt

Robots Exclusion Standard data for pornbay.tv

Archived Snapshots

Resource Scan

Scan Details

Site Domain	pornbay.tv
Base Domain	pornbay.tv
Scan Status	Ok
Last Scan	2024-09-08T02:08:36+00:00
Next Scan	2024-10-08T02:08:36+00:00

Last Scan

Scanned	2024-09-08T02:08:36+00:00
URL	https://pornbay.tv/robots.txt
Domain IPs	104.21.41.148, 172.67.147.168, 2606:4700:3033::6815:2994, 2606:4700:3035::ac43:93a8
Response IP	104.21.41.148
Found	Yes
Hash	22bb2d1e22406eb18fd734b40798524802146ba3b2bd2966edd3b23706b58a34
SimHash	4f1fc400eab8

Groups

*

Rule

Path

Disallow

/*.json$

Disallow

/*.js$

Disallow

/api/

Disallow

/e/

Disallow

/health/

Disallow

/_page-perf/

Disallow

/_player-perf/

Disallow

/_client-error/

msnbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

bingbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

yandexbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

baiduspider

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

sogou

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

semrushbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

Allow

/$

blexbot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

semrushbot-ba

Rule

Path

Disallow

/

semrushbot-si

Rule

Path

Disallow

/

semrushbot-swa

Rule

Path

Disallow

/

semrushbot-ct

Rule

Path

Disallow

/

semrushbot-bm

Rule

Path

Disallow

/

splitsignalbot

Rule

Path

Disallow

/

amazonbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

Back to top

Other Records

Field

Value

sitemap

https://pornbay.tv/sitemap.xml

Back to top

Warnings

2 invalid lines.

Back to top