Log In Sign Up

selectism.com
robots.txt

Robots Exclusion Standard data for selectism.com

Resource Scan

Scan Details

Site Domain selectism.com
Base Domain selectism.com
Scan Status Ok
Last Scan2025-10-31T18:56:37+00:00
Next Scan 2025-11-30T18:56:37+00:00

Last Scan

Scanned2025-10-31T18:56:37+00:00
URL https://selectism.com/robots.txt
Redirect https://www.highsnobiety.com/robots.txt
Redirect Domain www.highsnobiety.com
Redirect Base highsnobiety.com
Domain IPs 104.21.46.68, 172.67.136.14, 2606:4700:3035::6815:2e44, 2606:4700:3036::ac43:880e
Redirect IPs 104.19.158.49, 104.19.159.49, 2606:4700::6813:9e31, 2606:4700::6813:9f31
Response IP 104.19.159.49
Found Yes
Hash 6460e9e6793c483e30e395e7338f12e5cc324dfa15bf06c829b25986d0f6575f
SimHash 660c9252fbd4

Groups

googlebot
googlebot-image
pinterest
*

Rule Path
Disallow */shop/*?*brand=*
Disallow */shop/*?*category=*
Disallow */shop/*?*color=*
Disallow */shop/*?*price=*
Disallow */shop/*?*size=*
Disallow */shop/*?*sortBy=*
Disallow */search/*?*page=*
Disallow *?*context=*
Disallow *?*query=*
Disallow *?*limit=*
Disallow *?*queryID=*
Disallow *?*utm_*=*
Disallow /health-check
Disallow /l/amazon-pay-x-highsnobiety

Other Records

Field Value
crawl-delay 1

yandex
slurp
twitterbot

Rule Path
Disallow */shop/*?*brand=*
Disallow */shop/*?*category=*
Disallow */shop/*?*color=*
Disallow */shop/*?*price=*
Disallow */shop/*?*size=*
Disallow */shop/*?*sortBy=*
Disallow */search/*?*page=*
Disallow *?*context=*
Disallow *?*query=*
Disallow *?*limit=*
Disallow *?*queryID=*
Disallow *?*utm_*=*
Disallow /health-check
Disallow /l/amazon-pay-x-highsnobiety

Other Records

Field Value
crawl-delay 5

scrapy

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

jetty

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://highsnobiety.com/sitemap-content.xml
sitemap https://highsnobiety.com/sitemap-shop.xml
sitemap https://highsnobiety.com/sitemap-news.xml
Back to top