Log In Sign Up

somethingmassive.com
robots.txt

Robots Exclusion Standard data for somethingmassive.com

Resource Scan

Scan Details

Site Domain somethingmassive.com
Base Domain somethingmassive.com
Scan Status Ok
Last Scan2025-10-03T17:44:15+00:00
Next Scan 2025-11-02T17:44:15+00:00

Last Scan

Scanned2025-10-03T17:44:15+00:00
URL https://somethingmassive.com/robots.txt
Domain IPs 34.111.179.208
Response IP 34.111.179.208
Found Yes
Hash 9ee09b1d4033253fac05d5b2475b5a68ae8dbf434caa378ed659ebe162c419f3
SimHash 68d81330e648

Groups

*

Rule Path
Allow /
Allow /case-studies/
Allow /portfolio/
Allow /projects/
Allow /services/
Allow /about/
Allow /contact/
Allow /ai-ingest.json
Allow /llms.txt
Disallow /admin/
Disallow /api/conversations/
Disallow /api/upload*
Disallow /api/generate-ai-ingest
Disallow /uploads/
Disallow /server/
Disallow /scripts/
Disallow /temp/
Disallow /dev/
Disallow /*.log$
Disallow /*.tmp$
Disallow /teststater/
Disallow /teststate*/
Disallow /reelsmall*/
Disallow /how-to-market-to-n*/
Disallow /jennifer-brian*
Disallow /nutpods-dairy-free-success*
Allow /api/case-studies
Allow /api/content/
Allow /images/
Allow /videos/
Allow *.jpg
Allow *.jpeg
Allow *.png
Allow *.webp
Allow *.mp4
Allow *.svg

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Allow /case-studies/
Allow /portfolio/
Allow /services/
Allow /ai-ingest.json
Allow /llms.txt

claudebot

Rule Path
Allow /case-studies/
Allow /portfolio/
Allow /services/
Allow /ai-ingest.json
Allow /llms.txt

perplexitybot

Rule Path
Allow /case-studies/
Allow /portfolio/
Allow /services/
Allow /ai-ingest.json
Allow /llms.txt

chatgpt-user

Rule Path
Allow /case-studies/
Allow /portfolio/
Allow /services/
Allow /ai-ingest.json
Allow /llms.txt

claude-web

Rule Path
Allow /case-studies/
Allow /portfolio/
Allow /services/
Allow /ai-ingest.json
Allow /llms.txt
Back to top

Other Records

Field Value
sitemap https://www.somethingmassive.com/sitemap.xml
sitemap https://www.somethingmassive.com/ai-ingest.json
Back to top

Comments

  • Robots.txt for Something Massive
  • Creative advertising agency — AI-friendly version
  • Allow crawling of public creative and brand assets
  • Disallow internal/admin areas
  • Block problematic/test URLs that appeared in search results
  • Allow AI-friendly API endpoints
  • Still allow image/media assets for SEO and AI training
  • Crawl delay to preserve resources
  • Sitemap and structured content access
  • AI Content Discovery - Multiple access points
  • Main AI content index (primary)
  • https://www.somethingmassive.com/ai-ingest.json
  • Case studies data
  • https://www.somethingmassive.com/case-studies.json
  • Standard well-known endpoint for AI crawlers
  • https://www.somethingmassive.com/.well-known/ai-content
  • Explicit permission for AI crawlers
Back to top