condenasttraveler.com
robots.txt

Robots Exclusion Standard data for condenasttraveler.com

Resource Scan

Scan Details

Site Domain condenasttraveler.com
Base Domain condenasttraveler.com
Scan Status Ok
Last Scan2024-06-09T17:45:24+00:00
Next Scan 2024-06-16T17:45:24+00:00

Last Scan

Scanned2024-06-09T17:45:24+00:00
URL http://condenasttraveler.com/robots.txt
Redirect https://www.cntraveler.com/robots.txt
Redirect Domain www.cntraveler.com
Redirect Base cntraveler.com
Domain IPs 54.204.119.249
Redirect IPs 108.156.133.15, 108.156.133.20, 108.156.133.65, 108.156.133.84, 2600:9000:2755:5000:e:3fd5:4b00:93a1, 2600:9000:2755:b200:e:3fd5:4b00:93a1, 2600:9000:2755:c00:e:3fd5:4b00:93a1, 2600:9000:2755:c200:e:3fd5:4b00:93a1, 2600:9000:2755:c600:e:3fd5:4b00:93a1, 2600:9000:2755:d000:e:3fd5:4b00:93a1, 2600:9000:2755:d600:e:3fd5:4b00:93a1, 2600:9000:2755:f200:e:3fd5:4b00:93a1
Response IP 108.156.133.15
Found Yes
Hash 948673e49d0bad6c625655c3091b93d17cd765168954964e3fb7a5ca369357bf
SimHash 7614d165ce13

Groups

*

Rule Path
Disallow /user/
Disallow /preview/
Disallow /search
Disallow /newsletters/
Disallow /video/right_rail*
Disallow /*?redirectURL=
Disallow /*%7B%7Burl%7D%7D
Disallow /*%7B%7Burl%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /*%7B%7Bsection%7D%7D
Disallow /*%7B%7Bsection%7D%7D
Disallow /product/
Disallow /user-context?referrer
Disallow /services.min.js?_=

Other Records

Field Value
crawl-delay 30

google-inspectiontool

Rule Path
Allow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

Other Records

Field Value
sitemap https://www.cntraveler.com/sitemap.xml
sitemap https://www.cntraveler.com/branded-sitemap.xml
sitemap https://www.cntraveler.com/feed/google-latest-news/sitemap-google-news
sitemap https://www.cntraveler.com/contributors-sitemap.xml
sitemap https://www.cntraveler.com/categories-sitemap.xml
sitemap https://www.cntraveler.com/bundles-sitemap.xml
sitemap https://www.cntraveler.com/feed/rss

Comments

  • disallow /user/ as there are incoming links going to pages within the /user/ directory that can't be accessed.