Log In Sign Up

csc.com
robots.txt

Robots Exclusion Standard data for csc.com

Resource Scan

Scan Details

Site Domain csc.com
Base Domain csc.com
Scan Status Ok
Last Scan2025-07-10T20:46:28+00:00
Next Scan 2025-08-09T20:46:28+00:00

Last Scan

Scanned2025-07-10T20:46:28+00:00
URL https://csc.com/robots.txt
Redirect https://dxc.com/robots.txt?merger=true
Redirect Domain dxc.com
Redirect Base dxc.com
Domain IPs 34.212.245.99, 54.209.74.112
Redirect IPs 151.101.131.10, 151.101.195.10, 151.101.3.10, 151.101.67.10
Response IP 151.101.131.10
Found Yes
Hash 66cad7a7413221fc072a4c427606ed8e7bd4d5c1fd124823e57f1831dd506b1c
SimHash 40543377cd73

Groups

*

Rule Path
Allow /
Disallow */search?search=
Disallow */search$

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

exabot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://dxc.com/sitemap.xml
sitemap https://dxc.com/us/en.sitemap.newsroom-sitemap.xml
Back to top

Comments

  • OpenAI’s main web crawler for training GPT models
  • used for AI training
  • used for generative AI
  • TikTok
  • OpenAI’s bot for indexing web content for ChatGPT search
  • Identifies user requests from ChatGPT
  • Bot from Perplexity.ai to fetch real-time info with citations
  • Exa.ai’s crawler for dev-focused AI search
Back to top