Log In Sign Up

pstrax.com
robots.txt

Robots Exclusion Standard data for pstrax.com

Resource Scan

Scan Details

Site Domain pstrax.com
Base Domain pstrax.com
Scan Status Ok
Last Scan2025-12-13T11:03:07+00:00
Next Scan 2026-01-12T11:03:07+00:00

Last Scan

Scanned2025-12-13T11:03:07+00:00
URL https://pstrax.com/robots.txt
Domain IPs 104.26.6.77, 104.26.7.77, 172.67.73.35, 2606:4700:20::681a:64d, 2606:4700:20::681a:74d, 2606:4700:20::ac43:4923
Response IP 104.26.6.77
Found Yes
Hash de1d9195e26fb514469e318c3595489b77f8ed8840f3dfcbc07c6fbb1a48fa60
SimHash 369ac940e4a4

Groups

gptbot

Rule Path
Allow /
Disallow

anthropic-ai

Rule Path
Allow /
Disallow

google-extended

Rule Path
Allow /
Disallow

ccbot

Rule Path
Allow /
Disallow

cohere-ai

Rule Path
Allow /
Disallow

applebot-extended

Rule Path
Allow /
Disallow

oai-searchbot

Rule Path
Allow /
Disallow

chatgpt-user

Rule Path
Allow /
Disallow

chatgpt-user/2.0

Rule Path
Allow /
Disallow

claudebot

Rule Path
Allow /
Disallow

claude-web

Rule Path
Allow /
Disallow

perplexitybot

Rule Path
Allow /
Disallow

perplexity-user

Rule Path
Allow /
Disallow

mistralai-user

Rule Path
Allow /
Disallow

youbot

Rule Path
Allow /
Disallow

duckassistbot

Rule Path
Allow /
Disallow

googlebot

Rule Path
Allow /
Disallow

bingbot

Rule Path
Allow /
Disallow

applebot

Rule Path
Allow /
Disallow

amazonbot

Rule Path
Allow /
Disallow

facebookbot

Rule Path
Allow /
Disallow

meta-externalagent

Rule Path
Allow /
Disallow

linkedinbot

Rule Path
Allow /
Disallow

bytespider

Rule Path
Allow /
Disallow

ai2bot

Rule Path
Allow /
Disallow

diffbot

Rule Path
Allow /
Disallow

omgili

Rule Path
Allow /
Disallow

timpibot

Rule Path
Allow /
Disallow

*

Rule Path
Allow /
Disallow
Back to top

Other Records

Field Value
sitemap https://pstrax.com/sitemap_index.xml
sitemap https://pstrax.com/sitemap.xml
Back to top

Comments

  • PSTrax Robots.txt
  • Last Updated: July 2025
  • ===========================================
  • AI MODEL TRAINING BOTS
  • ===========================================
  • These bots collect data for training AI models.
  • ===========================================
  • REAL-TIME AI SEARCH & CITATION BOTS
  • ===========================================
  • These bots fetch content when users interact with AI.
  • ===========================================
  • TRADITIONAL SEARCH ENGINES WITH AI
  • ===========================================
  • ===========================================
  • SOCIAL MEDIA AI CRAWLERS
  • ===========================================
  • ===========================================
  • SPECIALIZED AI CRAWLERS
  • ===========================================
  • ===========================================
  • DEFAULT RULES FOR ALL OTHER BOTS
  • ===========================================
  • ===========================================
  • SITEMAP REFERENCES
  • ===========================================
Back to top