agentcrossy.neocities.org
robots.txt

Robots Exclusion Standard data for agentcrossy.neocities.org

Resource Scan

Scanned	2025-10-30T02:35:25+00:00
URL	https://agentcrossy.neocities.org/robots.txt
Domain IPs	23.143.32.16, 2620:2:6000::a:1
Response IP	23.143.32.16
Found	Yes
Hash	2b85337f5dba30258025178e1be6addcbacf48547d5eac9a17a9f1c9e10d9a90
SimHash	73160901c7e5

Rule	Path
Allow	/

Rule

Path

Allow

/

Back to top

This file tells search engines and bots what they are allowed to see on your site.
This is the default rule, which allows search engines to crawl your site (recommended).
If you do not want AI bots to crawl your site, remove the # from the following lines:
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
User-agent: Sidetrade indexer bot
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
Disallow: /

Back to top