Log In Sign Up

goodspace.ai
robots.txt

Robots Exclusion Standard data for goodspace.ai

Resource Scan

Scan Details

Site Domain goodspace.ai
Base Domain goodspace.ai
Scan Status Ok
Last Scan2025-12-13T02:30:17+00:00
Next Scan 2025-12-20T02:30:17+00:00

Last Scan

Scanned2025-12-13T02:30:17+00:00
URL https://goodspace.ai/robots.txt
Domain IPs 34.131.200.225
Response IP 34.131.200.225
Found Yes
Hash c25f9f9ac2e7ac8066d19e8ad6269467db3e27ef79eb232fee30add2ed83474c
SimHash 75b39371aff0

Groups

*

Rule Path
Allow /
Disallow /create-job-post/
Disallow /auth/
Disallow /login
Disallow /logout
Disallow /register
Disallow /reset-password
Disallow /verify/
Disallow /dashboard/
Disallow /account/
Disallow /settings/
Disallow /onboarding/
Disallow /payment/
Disallow /checkout/
Disallow /api/
Disallow /admin/
Disallow /users/
Allow /users/featured/

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

claudebot

Rule Path
Allow /

google-extended

Rule Path
Allow /

bingchatbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

youbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

telegram

Rule Path
Allow /

slackbot

Rule Path
Allow /

discordbot

Rule Path
Allow /

pinterest

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

redditbot

Rule Path
Allow /

tumblr

Rule Path
Allow /

skypeuripreview

Rule Path
Allow /

flipboard

Rule Path
Allow /

vkshare

Rule Path
Allow /

viber

Rule Path
Allow /

kakaotalk

Rule Path
Allow /

wechat

Rule Path
Allow /

line

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

majestic

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

serpstatbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

screaming frog seo spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

gtmetrix

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

pingdom

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

site24x7

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

blexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

seostar

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

alexabot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

sogou

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

naver

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

seznambot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

mail.ru

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

daumoa

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

qwantify

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

petalsearch

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

w3c_validator

Rule Path
Allow /

chrome-lighthouse

Rule Path
Allow /

google-inspectiontool

Rule Path
Allow /

google page speed

Rule Path
Allow /

pagespeed

Rule Path
Allow /

lighthouse

Rule Path
Allow /

ptst

Rule Path
Allow /

validator.nu

Rule Path
Allow /

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

apple-applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

amazonbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

alexabot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

rogerbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

embedly

Rule Path
Allow /

quora link preview

Rule Path
Allow /

showyoubot

Rule Path
Allow /

outbrain

Rule Path
Allow /

developers.google.com/+/web/snippet

Rule Path
Allow /

nuzzel

Rule Path
Allow /

bitlybot

Rule Path
Allow /

bitrix link preview

Rule Path
Allow /

xing-contenttabreceiver

Rule Path
Allow /

integration-test

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://goodspace.ai/sitemap_index.xml
Back to top

Comments

  • Robots.txt for GoodSpace.ai
  • Updated: 2025-11-16
  • Synchronized with Cloudflare Worker bot detection for prerendering
  • ============================================
  • DEFAULT ACCESS FOR ALL CRAWLERS
  • ============================================
  • Block utility pages and flows
  • Block user profiles (except curated ones)
  • ============================================
  • TRADITIONAL SEARCH ENGINES - ALLOW WITH CRAWL DELAY
  • ============================================
  • ============================================
  • AI/LLM CRAWLERS - ALLOW FOR AI-POWERED SEARCH & DISCOVERY
  • ============================================
  • OpenAI Bots
  • Anthropic Bots
  • Google AI Bots
  • Microsoft AI Bots
  • Perplexity
  • Other AI Crawlers
  • ============================================
  • SOCIAL MEDIA BOTS - ALLOW FOR LINK PREVIEWS
  • ============================================
  • ============================================
  • SEO & ANALYTICS TOOLS - CONTROLLED ACCESS
  • ============================================
  • ============================================
  • REGIONAL SEARCH ENGINES
  • ============================================
  • ============================================
  • DEVELOPER TOOLS & VALIDATORS
  • ============================================
  • ============================================
  • APPLE & AMAZON SERVICES
  • ============================================
  • ============================================
  • OTHER IMPORTANT BOTS
  • ============================================
  • ============================================
  • SITEMAP
  • ============================================
Back to top

Warnings

  • 3 invalid lines.
Back to top