Log In Sign Up

aifreeforever.com
robots.txt

Robots Exclusion Standard data for aifreeforever.com

Resource Scan

Scan Details

Site Domain aifreeforever.com
Base Domain aifreeforever.com
Scan Status Ok
Last Scan2025-09-15T10:45:23+00:00
Next Scan 2025-09-22T10:45:23+00:00

Last Scan

Scanned2025-09-15T10:45:23+00:00
URL https://aifreeforever.com/robots.txt
Domain IPs 104.21.19.177, 172.67.187.9, 2606:4700:3033::ac43:bb09, 2606:4700:3036::6815:13b1
Response IP 172.67.187.9
Found Yes
Hash 44099fd13a0d4e6a99a78959abecf2736d7fa77381e077523e1fdc8026e4b787
SimHash 3c189553e7d2

Groups

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

yandexbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

chatgpt-user

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

openai-webcrawler

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

claude-web

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

claudebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

youbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

msedge

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingpreview

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

meta-externalagent

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

coherebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

mistralbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

huggingfacebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

stabilitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

replicatebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

characterai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

jasperbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

copyai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

writesonicbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

grammarlybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

notionbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

canvabot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

adobebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

midjourneybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

dallebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

stablediffusionbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

scholarbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

researchbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

academicbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

discordbot

Rule Path
Allow /

slackbot

Rule Path
Allow /

redditbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

mozbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

majesticseo

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

uptimerobot

Rule Path
Allow /

pingdom

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

wayback

Rule Path
Allow /

feedlybot

Rule Path
Allow /

flipboardproxy

Rule Path
Allow /
Allow /sitemap-0.xml
Allow /robots.txt
Allow /favicon.ico
Allow /manifest.json
Allow /*/
Allow /*.css
Allow /*.js
Allow /*.png
Allow /*.jpg
Allow /*.jpeg
Allow /*.gif
Allow /*.webp
Allow /*.svg
Allow /*.ico
Allow /*.woff
Allow /*.woff2
Allow /*.ttf
Allow /*.eot
Allow /*.pdf
Allow /*.txt
Allow /*.xml
Allow /*.json
Back to top

Other Records

Field Value
sitemap https://aifreeforever.com/sitemap-0.xml
Back to top

Comments

  • NOTICE: The collection of content and other data on this
  • site through automated means, including any device, tool,
  • or process designed to data mine or scrape content, is
  • prohibited except (1) for the purpose of search engine indexing or
  • artificial intelligence retrieval augmented generation or (2) with express
  • written permission from this site’s operator.
  • To request permission to license our intellectual
  • property and/or other materials, please contact this
  • site’s operator directly.
  • BEGIN Cloudflare Managed content
  • END Cloudflare Managed Content
  • Robots.txt for AI Free Forever - https://aifreeforever.com
  • Allow all search engines and AI crawlers to access the site
  • Default rule for all crawlers
  • Google Search Engine
  • Google Images
  • Google News
  • Google Mobile
  • Bing Search Engine
  • Yahoo Search Engine
  • Yandex Search Engine
  • Baidu Search Engine
  • DuckDuckGo Search Engine
  • AI Search Engines and Crawlers
  • OpenAI GPTBot
  • ChatGPT User Agent
  • OpenAI WebCrawler
  • Anthropic Claude
  • Anthropic ClaudeBot
  • Perplexity AI
  • You.com AI Search
  • Microsoft Copilot
  • Bing AI Chat
  • Facebook/Meta AI
  • Meta AI
  • X (Twitter) AI
  • LinkedIn AI
  • Apple Intelligence
  • Cohere AI
  • Mistral AI
  • Hugging Face
  • Stability AI
  • Replicate AI
  • Character.AI
  • Jasper AI
  • Copy.ai
  • Writesonic
  • Grammarly AI
  • Notion AI
  • Canva AI
  • Adobe AI
  • Midjourney (if they have a crawler)
  • DALL-E related crawlers
  • Stable Diffusion crawlers
  • Academic and Research AI Crawlers
  • Social Media Platform Crawlers
  • SEO and Analytics Tools
  • Archive Crawlers
  • RSS and Feed Readers
  • Disallow sensitive areas (if any exist)
  • Uncomment and modify as needed
  • Disallow: /admin/
  • Disallow: /private/
  • Disallow: /api/internal/
  • Disallow: /.env
  • Disallow: /config/
  • Allow important files
  • Sitemap location
  • Clean URLs - Allow trailing slashes
  • Allow common file types
  • Last updated: 2025-01-11
  • This robots.txt file is designed to be AI-friendly and supports all major search engines
  • and AI crawlers to help with content discovery and indexing
Back to top

Warnings

  • 1 invalid line.
  • `host` is not a known field.
Back to top