/.well-known/

Log In Sign Up

pornpig.tube
robots.txt

Robots Exclusion Standard data for pornpig.tube

Archived Snapshots

Resource Scan

Scan Details

Site Domain	pornpig.tube
Base Domain	pornpig.tube
Scan Status	Ok
Last Scan	2024-09-19T08:26:06+00:00
Next Scan	2024-10-19T08:26:06+00:00

Last Scan

Scanned	2024-09-19T08:26:06+00:00
URL	https://pornpig.tube/robots.txt
Domain IPs	104.21.30.79, 172.67.172.128, 2606:4700:3031::ac43:ac80, 2606:4700:3035::6815:1e4f
Response IP	104.21.30.79
Found	Yes
Hash	2873aebef9ceb7ac8e9251a52b90905839503ce07b73fdb7e48119351431687e
SimHash	4f1fc402cab8

Groups

*

Rule

Path

Disallow

/*.json$

Disallow

/*.js$

Disallow

/api/

Disallow

/e/

Disallow

/health/

Disallow

/_page-perf/

Disallow

/_player-perf/

Disallow

/_client-error/

msnbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

bingbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

2

yandexbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

baiduspider

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

sogou

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

3

semrushbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

Allow

/$

blexbot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

semrushbot-ba

Rule

Path

Disallow

/

semrushbot-si

Rule

Path

Disallow

/

semrushbot-swa

Rule

Path

Disallow

/

semrushbot-ct

Rule

Path

Disallow

/

semrushbot-bm

Rule

Path

Disallow

/

splitsignalbot

Rule

Path

Disallow

/

amazonbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

Back to top

Other Records

Field

Value

sitemap

https://pornpig.tube/sitemap.xml

Back to top

Warnings

2 invalid lines.

Back to top