blu-ray.com
robots.txt

Robots Exclusion Standard data for blu-ray.com

Resource Scan

Scan Details

Site Domain blu-ray.com
Base Domain blu-ray.com
Scan Status Ok
Last Scan2024-11-16T09:50:28+00:00
Next Scan 2024-11-23T09:50:28+00:00

Last Scan

Scanned2024-11-16T09:50:28+00:00
URL https://blu-ray.com/robots.txt
Redirect https://www.blu-ray.com/robots.txt
Redirect Domain www.blu-ray.com
Redirect Base blu-ray.com
Domain IPs 64.21.0.130
Redirect IPs 64.21.0.130
Response IP 64.21.0.130
Found Yes
Hash 326d9b53658112d19cf980063b41338bc198fb9b29671ab24e1073fc25fdb825
SimHash 5034c240f601

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /community/collection.php
Disallow /community/favorites.php
Disallow /community/profile.php
Disallow /community/ratings.php
Disallow /link/
Disallow /movies/search.php
Disallow /news/search.php
Disallow /search/

mediapartners-google

Rule Path
Allow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

proximic

Rule Path
Disallow /

riddler

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

maxpointcrawler/nutch-1.10

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

seekport

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ntentbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

thesis-research-bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /