discordguildas.neocities.org
robots.txt

Robots Exclusion Standard data for discordguildas.neocities.org

Resource Scan

Site Domain	discordguildas.neocities.org
Base Domain	neocities.org
Scan Status	Ok
Last Scan	2025-06-02T08:07:47+00:00
Next Scan	2025-07-02T08:07:47+00:00

Scanned	2025-06-02T08:07:47+00:00
URL	https://discordguildas.neocities.org/robots.txt
Domain IPs	198.51.233.2, 2620:2:6000::a:1
Response IP	198.51.233.2
Found	Yes
Hash	2b85337f5dba30258025178e1be6addcbacf48547d5eac9a17a9f1c9e10d9a90
SimHash	73160901c7e5

Rule	Path
Allow	/

Rule

Path

Allow

/

Back to top

This file tells search engines and bots what they are allowed to see on your site.
This is the default rule, which allows search engines to crawl your site (recommended).
If you do not want AI bots to crawl your site, remove the # from the following lines:
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
User-agent: Sidetrade indexer bot
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
Disallow: /

Back to top