Log In Sign Up

vanityfair.co.uk
robots.txt

Robots Exclusion Standard data for vanityfair.co.uk

Resource Scan

Scan Details

Site Domain vanityfair.co.uk
Base Domain vanityfair.co.uk
Scan Status Ok
Last Scan2024-11-11T01:56:36+00:00
Next Scan 2024-11-18T01:56:36+00:00

Last Scan

Scanned2024-11-11T01:56:36+00:00
URL https://vanityfair.co.uk/robots.txt
Redirect https://www.vanityfair.com/robots.txt
Redirect Domain www.vanityfair.com
Redirect Base vanityfair.com
Domain IPs 2a05:d018:f0d:d700:c34e:8388:1340:84ee, 2a05:d018:f0d:d701:8e83:49d1:24f8:962f, 2a05:d018:f0d:d702:52bf:70b:5ab8:97c6, 34.242.143.176, 34.247.10.29, 52.31.100.167
Redirect IPs 13.33.183.12, 13.33.183.45, 13.33.183.53, 13.33.183.62, 2600:9000:2078:1a00:f:ad27:1740:93a1, 2600:9000:2078:2a00:f:ad27:1740:93a1, 2600:9000:2078:3e00:f:ad27:1740:93a1, 2600:9000:2078:4800:f:ad27:1740:93a1, 2600:9000:2078:5200:f:ad27:1740:93a1, 2600:9000:2078:7800:f:ad27:1740:93a1, 2600:9000:2078:a800:f:ad27:1740:93a1, 2600:9000:2078:c00:f:ad27:1740:93a1
Response IP 13.35.210.57
Found Yes
Hash ac411faa3de5354614b7de4da9cd730afb9a7c71cac27fd32dadf5f3a8820d7e
SimHash 441ac153f682

Groups

*

Rule Path
Disallow /*/amp
Disallow /api/
Disallow /data/
Disallow /user/
Disallow /ontheweb/blogs/delineator
Disallow /preview/
Disallow /search?
Disallow /search/
Disallow /video/right_rail*
Disallow /*?redirectURL=
Disallow /*%7B%7Burl%7D%7D
Disallow /*%7B%7Burl%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /*.pdf
Disallow /*%7B%7Bsection%7D%7D
Disallow /*%7B%7Bsection%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /product/
Disallow /journey/compiler/amp/iframe.html
Disallow /api/search?page=
Disallow /user-context?referrer
Disallow /services.min.js?_=
Disallow /com.condenast/yv8*

googlebot-news

Rule Path
Disallow /branded/
Allow /*/amp
Disallow /search?
Disallow /search/
Disallow /*?redirectURL=
Disallow /*%7B%7Burl%7D%7D
Disallow /*%7B%7Burl%7D%7D
Disallow /*.pdf
Disallow /*%7B%7Bsection%7D%7D
Disallow /*%7B%7Bsection%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /product/
Disallow /journey/compiler/amp/iframe.html
Disallow /api/search?page=
Disallow /user-context?referrer
Disallow /services.min.js?_=

googlebot

Rule Path
Allow /*/amp
Disallow /search?
Disallow /search/
Disallow /*?redirectURL=
Disallow /*%7B%7Burl%7D%7D
Disallow /*%7B%7Burl%7D%7D
Disallow /*.pdf
Disallow /*%7B%7Bsection%7D%7D
Disallow /*%7B%7Bsection%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /product/
Disallow /journey/compiler/amp/iframe.html
Disallow /api/search?page=
Disallow /user-context?referrer
Disallow /services.min.js?_=
Disallow /preview/

bingbot

Rule Path
Allow /*/amp
Disallow /*?redirectURL=
Disallow /search?
Disallow /search/
Disallow /*%7B%7Burl%7D%7D
Disallow /*%7B%7Burl%7D%7D
Disallow /*.pdf
Disallow /*%7B%7Bsection%7D%7D
Disallow /*%7B%7Bsection%7D%7D
Disallow /auth/*
Disallow /account/*
Disallow /product/
Disallow /journey/compiler/amp/iframe.html
Disallow /api/search?page=
Disallow /user-context?referrer
Disallow /services.min.js?_=

google-inspectiontool

Rule Path
Allow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

proximic

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

heritrix

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

cliqzbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120
Back to top

Other Records

Field Value
sitemap https://www.vanityfair.com/sitemap.xml
sitemap https://www.vanityfair.com/branded-sitemap.xml
sitemap https://www.vanityfair.com/sitemap/people.xml
sitemap https://www.vanityfair.com/feed/google-latest-news/sitemap-google-news
sitemap https://www.vanityfair.com/categories-sitemap.xml
sitemap https://www.vanityfair.com/contributors-sitemap.xml
sitemap https://www.vanityfair.com/bundles-sitemap.xml
sitemap https://www.vanityfair.com/sitemap-archive-1.xml
Back to top