Log In Sign Up

harpersbazaar.com.sg
robots.txt

Robots Exclusion Standard data for harpersbazaar.com.sg

Resource Scan

Scan Details

Site Domain harpersbazaar.com.sg
Base Domain harpersbazaar.com.sg
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-28T13:08:42+00:00
Next Scan 2025-01-26T13:08:42+00:00

Last Successful Scan

Scanned2024-07-01T13:06:28+00:00
URL https://harpersbazaar.com.sg/robots.txt
Redirect https://www.harpersbazaar.com.sg/robots.txt
Redirect Domain www.harpersbazaar.com.sg
Redirect Base harpersbazaar.com.sg
Domain IPs 108.156.133.115, 108.156.133.66, 108.156.133.8, 108.156.133.90
Redirect IPs 18.67.181.46, 18.67.181.52, 18.67.181.70, 18.67.181.92
Response IP 13.35.18.82
Found Yes
Hash 2036804ae75e814c61a605bac2d077ee12f9e280aca456769d6ba4eecfe98d79
SimHash d0589904fa33

Groups

*

Rule Path
Disallow /*/feed/$
Disallow /advanced-galleries/*
Disallow /search/*
Disallow /topics/

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.harpersbazaar.com.sg/_plat/api/sitemap.xml
Back to top

Comments

  • For new training only
  • Not for training, only for user requests
  • Marker for disabling Bard and Vertex AI
  • Multi-purpose, commercial uses; including LLMs
Back to top