Log In Sign Up

starfish-studios.com
robots.txt

Robots Exclusion Standard data for starfish-studios.com

Resource Scan

Scan Details

Site Domain starfish-studios.com
Base Domain starfish-studios.com
Scan Status Ok
Last Scan2025-12-16T17:42:28+00:00
Next Scan 2025-12-30T17:42:28+00:00

Last Scan

Scanned2025-12-16T17:42:28+00:00
URL https://starfish-studios.com/robots.txt
Domain IPs 2a10:9840:1:7200::, 2a10:9840:2:7300::, 31.43.172.0, 31.43.173.0
Response IP 31.43.172.0
Found Yes
Hash c2b16665e1e68455474914d533140a6d620069885a30ab8f5e940a0b3b4c8b84
SimHash 181d9912df73

Groups

archivebot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

qwantbot

Rule Path
Allow *

panascais-search-bot

Rule Path
Allow *

googlebot
googlebot-image
googlebot-news
googlebot-video
googlebot-mobile
google-inspectiontool
storebot-google

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

yandexbot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

yeti

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

qwantify

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

ecosiabot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

startpagebot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

searx

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

kagibot

Rule Path
Allow *

Other Records

Field Value
crawl-delay 4

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot
chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot
siteauditbot
splitsignalbot
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ocob
semrushbot-ft

Rule Path
Disallow /

dotbot
rogerbot

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

penthesilea

Rule Path
Disallow /

*

Rule Path
Allow *
Back to top

Other Records

Field Value
sitemap https://starfish-studios.com/sitemap.xml
Back to top

Comments

  • ===================================
  • Website: https://starfish-studios.com/
  • ===================================
  • ===================================
  • Allowed Crawlers
  • ===================================
  • Archive.org Configuration
  • Qwant Configuration
  • Panascais Search Configuration
  • Google Configuration
  • Bing Configuration
  • Yahoo! Configuration
  • DuckDuckGo Configuration
  • Yandex Configuration
  • Baidu Configuration
  • Naver Configuration
  • Qwant Configuration
  • Ecosia Configuration
  • Startpage Configuration
  • Searx Configuration
  • Kagi Configuration
  • ===================================
  • Disallowed Crawlers
  • ===================================
  • Google Extended Configuration
  • Common Crawl Configuration
  • OpenAI Configuration
  • Perplexity AI Configuration
  • Majestic 12 SEO Configuration
  • Ahrefs Configuration
  • SEMrush Configuration
  • SEOmoz Configuration
  • netEstate NE Configuration
  • Geedo Product Search Configuration
  • ===================================
  • Generic Bad Crawlers
  • ===================================
  • ===================================
  • Global Configuration
  • ===================================
Back to top