Log In Sign Up

linkster.co
robots.txt

Robots Exclusion Standard data for linkster.co

Resource Scan

Scan Details

Site Domain linkster.co
Base Domain linkster.co
Scan Status Ok
Last Scan2025-12-15T07:51:33+00:00
Next Scan 2026-01-14T07:51:33+00:00

Last Scan

Scanned2025-12-15T07:51:33+00:00
URL https://linkster.co/robots.txt
Redirect https://www.linkster.co/robots.txt
Redirect Domain www.linkster.co
Redirect Base linkster.co
Domain IPs 2a01:488:42:1000:50ed:8242:fffb:706f, 5.175.14.66
Redirect IPs 2600:9000:2894:2a00:19:6c08:540:93a1, 2600:9000:2894:400:19:6c08:540:93a1, 2600:9000:2894:7800:19:6c08:540:93a1, 2600:9000:2894:dc00:19:6c08:540:93a1, 2600:9000:2894:e000:19:6c08:540:93a1, 2600:9000:2894:e400:19:6c08:540:93a1, 2600:9000:2894:e800:19:6c08:540:93a1, 2600:9000:2894:fc00:19:6c08:540:93a1, 3.170.229.19, 3.170.229.23, 3.170.229.69, 3.170.229.77
Response IP 3.170.229.23
Found Yes
Hash e000298052f0bc693459cd37ae385af7070f15d6dd2d4ba5acbf29384253d69b
SimHash 28149ad2e781

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /api/
Disallow /_astro/
Allow /de/
Allow /*.js$
Allow /*.css$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Allow /*.woff2$

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.linkster.co/sitemap-index.xml
Back to top

Comments

  • Linkster Robots.txt
  • Sitemaps
  • Crawl-delay for respectful crawling
  • Disallow access to internal areas
  • Allow search engines to access everything else
  • Specific user agents
  • Block bad bots
Back to top