subtitlestar.com
robots.txt
Robots Exclusion Standard data for subtitlestar.com
Resource Scan
Scan Details
| Site Domain | subtitlestar.com |
| Base Domain | subtitlestar.com |
| Scan Status | Ok |
| Last Scan | 2025-08-31T06:06:55+00:00 |
| Next Scan | 2025-09-30T06:06:55+00:00 |
Last Scan
| Scanned | 2025-08-31T06:06:55+00:00 |
| URL | https://subtitlestar.com/robots.txt |
| Domain IPs | 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001 |
| Response IP | 104.21.80.1 |
| Found | Yes |
| Hash | 809a0e9150630cf60817398464dc3e04ad5d49a9dc98a2593438a229af24838d |
| SimHash | 7a1d5851c4e0 |
Groups
ahrefsbot
mj12bot
dataforseobot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 100 |
yandex
semrushbot
semrushbot-sa
blexbot
turnitinbot
heritrix
barkrowler
adsbot
awariorssbot
awariosmartbot
| Rule | Path |
|---|---|
| Disallow | / |
chatgpt-user
claude-web
meta-externalfetcher
oai-searchbot
applebot
applebot-extended
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 80 |
meta-externalagent
claudebot
gptbot
ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
brightbot 1.0
bytespider
ccbot
cohere-ai
cohere-training-data-crawler
crawlspace
diffbot
duckassistbot
facebookbot
friendlycrawler
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
omgili
omgilibot
pangubot
perplexitybot
perplexityâuser
petalbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /wp-admin/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://subtitlestar.com/sitemap_index.xml |
Comments