Log In Sign Up

headcycle.com
robots.txt

Robots Exclusion Standard data for headcycle.com

Resource Scan

Scan Details

Site Domain headcycle.com
Base Domain headcycle.com
Scan Status Ok
Last Scan2025-10-14T21:31:33+00:00
Next Scan 2025-11-13T21:31:33+00:00

Last Scan

Scanned2025-10-14T21:31:33+00:00
URL https://headcycle.com/robots.txt
Redirect https://www.headcycle.com/robots.txt
Redirect Domain www.headcycle.com
Redirect Base headcycle.com
Domain IPs 13.248.132.87, 35.71.145.101, 75.2.97.79, 99.83.151.71
Redirect IPs 13.248.132.87, 35.71.145.101, 75.2.97.79, 99.83.151.71
Response IP 35.71.145.101
Found Yes
Hash 09fe05087499cbd23a8fe30319c3d237b85c664b8d96eabc766629b1349a3eea
SimHash 525fd148fa51

Groups

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

serpstatbot/2.1

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

femtosearchbot/1.0

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dataforseobot/1.0

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

my-tiny-bot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

friendlycrawler/1.0

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

facebookcatalog/1.0

Rule Path
Disallow /

user-agent: velenpublicwebcrawler

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdcbot/1.0

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

dataforseobot/1.0

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-searchbot/1.0

Rule Path
Disallow /

*

Rule Path
Disallow /login
Disallow /sign_up
Disallow /wordcloud
Disallow /wordcloud2
Disallow /h/*/new
Disallow /h/*/wordcloud
Disallow /domain/*/new
Disallow /domain/*/wordcloud
Disallow /user/*/new
Disallow /user/*/comments
Disallow /user/*/submitted
Disallow /user/*/wordcloud
Disallow /comment_share/*
Back to top

Comments

  • Disallow: /rss
  • Disallow: /search
  • Disallow: /h/*/search
  • Disallow: /h/*/rss
  • Disallow: /domain/*/search
  • Disallow: /user/*/search
  • this works, but the /search rule above actually works first
  • Disallow: /*?q=
  • no idea if this does the same thing or is more acurate
  • Disallow: /*?q=*
  • Disallow: /*?after=
  • Disallow: /*?before=
  • Disallow all query strings - might not be the best solution
  • Disallow: /*?*
Back to top