Log In Sign Up

fl20230831.fastlink.la
robots.txt

Robots Exclusion Standard data for fl20230831.fastlink.la

Resource Scan

Scan Details

Site Domain fl20230831.fastlink.la
Base Domain fastlink.la
Scan Status Ok
Last Scan2025-10-20T15:33:41+00:00
Next Scan 2025-11-19T15:33:41+00:00

Last Scan

Scanned2025-10-20T15:33:41+00:00
URL https://fl20230831.fastlink.la/robots.txt
Domain IPs 104.21.50.242, 172.67.215.2, 2606:4700:3032::6815:32f2, 2606:4700:3034::ac43:d702
Response IP 104.21.50.242
Found Yes
Hash 90ec65235e62d33c61025740ff30c99f7e687fb9754c11b0ce2aea48eca9e4dd
SimHash 2951f2f0955f

Groups

gptbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bingbot

Rule Path
Allow /

bingpreview

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

duckduckbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Disallow /

facebookcatalog

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Comments

  • OpenAI
  • Google AI 扩展（非 Googlebot）
  • Bing AI
  • Claude（Anthropic）
  • Perplexity
  • You.com
  • Amazon AI
  • DuckDuckGo AI
  • Facebook (AI-related scrapers)
  • Archive.org
  • å
Back to top

Warnings

  • 1 invalid line.
Back to top