Log In Sign Up

hdasianporntube.com
robots.txt

Robots Exclusion Standard data for hdasianporntube.com

Resource Scan

Scan Details

Site Domain hdasianporntube.com
Base Domain hdasianporntube.com
Scan Status Ok
Last Scan2024-09-15T10:58:13+00:00
Next Scan 2024-10-15T10:58:13+00:00

Last Scan

Scanned2024-09-15T10:58:13+00:00
URL https://hdasianporntube.com/robots.txt
Domain IPs 104.21.233.243, 104.21.233.244, 2606:4700:3038::6815:e9f3, 2606:4700:3038::6815:e9f4
Response IP 104.21.233.243
Found Yes
Hash 93e324b8cd3ad3d8a2017e502a191ef26f2b342cc441d9e5d7ced5cd319a8670
SimHash 506ec088e579

Groups

yandexvideoparser

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mail.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

yandex

Rule Path
Disallow /de/
Disallow /fr/
Disallow /it/
Disallow /es/
Disallow /pt/
Disallow /nl/
Disallow /sv/
Disallow /id/
Disallow /ms/
Disallow /ur/
Disallow /ja/
Disallow /ko/
Disallow /th/
Disallow /hi/
Disallow /zh/
Disallow /ar/
Disallow /tags/de/
Disallow /tags/fr/
Disallow /tags/it/
Disallow /tags/es/
Disallow /tags/pt/
Disallow /tags/nl/
Disallow /tags/sv/
Disallow /tags/id/
Disallow /tags/ms/
Disallow /tags/ur/
Disallow /tags/ja/
Disallow /tags/ko/
Disallow /tags/th/
Disallow /tags/hi/
Disallow /tags/zh/
Disallow /tags/ar/
Back to top

Warnings

  • 2 invalid lines.
Back to top