Log In Sign Up

towardsdatascience.com
robots.txt

Robots Exclusion Standard data for towardsdatascience.com

Resource Scan

Scan Details

Site Domain towardsdatascience.com
Base Domain towardsdatascience.com
Scan Status Ok
Last Scan2025-08-28T08:53:37+00:00
Next Scan 2025-09-11T08:53:37+00:00

Last Scan

Scanned2025-08-28T08:53:37+00:00
URL https://towardsdatascience.com/robots.txt
Domain IPs 141.193.213.20, 141.193.213.21
Response IP 141.193.213.20
Found Yes
Hash c86304e7843fa68c7a026220e14ceb331fde3192d91ee595222c23d2686e185c
SimHash 411c4940e093

Groups

*

Rule Path
Disallow

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

seekr

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://towardsdatascience.com/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
Back to top

Warnings

  • 1 invalid line.
Back to top