Log In Sign Up

craigswapp.com
robots.txt

Robots Exclusion Standard data for craigswapp.com

Resource Scan

Scan Details

Site Domain craigswapp.com
Base Domain craigswapp.com
Scan Status Ok
Last Scan2025-12-13T15:31:30+00:00
Next Scan 2026-01-12T15:31:30+00:00

Last Scan

Scanned2025-12-13T15:31:30+00:00
URL https://craigswapp.com/robots.txt
Redirect https://www.craigswapp.com/robots.txt
Redirect Domain www.craigswapp.com
Redirect Base craigswapp.com
Domain IPs 194.1.147.14, 194.1.147.7
Redirect IPs 194.1.147.14, 194.1.147.7
Response IP 194.1.147.14
Found Yes
Hash 9c242e6d07a2feba3d724b7f8522e645d4c4033d7efe969849724ab0f8e1b61a
SimHash 72909913a1e7

Groups

*

Rule Path
Disallow /wp-admin/
Disallow */feed
Allow /wp-admin/admin-ajax.php

*

Rule Path
Allow /

gptbot
chatgpt-user
oai-searchbot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

google-extended
gemini-pro
googlebot-ai
googlebot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

claudebot
claude-web
anthropic-ai

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

perplexitybot
perplexity-ai

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

meta-externalagent
meta-ai
facebookbot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

bingbot
bing-ai
copilotbot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

youbot
duckassistbot
bravebot
kogibot
arcbot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

ccbot

Rule Path
Allow /
Allow /.well-known/ai-plugin.json
Allow /.well-known/openapi.json

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.craigswapp.com/sitemap_index.xml
sitemap https://www.craigswapp.com/ai-sitemap.xml
Back to top

Comments

  • This file is automatically added by Rank Math SEO plugin to help a website index better
  • More info: https://rankmath.com/?utm_source=Plugin&utm_medium=Robots&utm_campaign=WP
  • ---- LightSite.ai AI Discovery Rules (October 2025) ----
  • OpenAI Crawlers (2025)
  • Google AI Crawlers (2025)
  • Anthropic Claude (2025)
  • Perplexity AI (2025)
  • Meta AI (2025)
  • Microsoft Copilot (2025)
  • Other AI Platforms (2025)
  • Common Crawlers
  • Crawl Rate
  • AI Discovery Sitemaps
  • ---- LightSite.ai AI Discovery Rules END ----
Back to top