slate.com
robots.txt

Robots Exclusion Standard data for slate.com

Resource Scan

Scan Details

Site Domain slate.com
Base Domain slate.com
Scan Status Ok
Last Scan2024-11-02T10:41:13+00:00
Next Scan 2024-11-09T10:41:13+00:00

Last Scan

Scanned2024-11-02T10:41:13+00:00
URL https://slate.com/robots.txt
Domain IPs 151.101.130.133, 151.101.194.133, 151.101.2.133, 151.101.66.133
Response IP 151.101.2.133
Found Yes
Hash 9864a16b54a79c27ea100fcca026cfa9ee65a1fcb4b1e7c0bab84134c6f32102
SimHash 4c3459506494

Groups

mediapartners-google

Rule Path
Disallow

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /search
Disallow /comments/

Other Records

Field Value
crawl-delay 10

mediapartners-google*

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /css
Disallow /fonts
Disallow /media
Disallow /piano
Disallow /static

Other Records

Field Value
crawl-delay 10

anthropic-ai
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
facebookbot
friendlycrawler
google-extended
googleother
gptbot
img2dataset
imagesiftbot
magpie-crawler
meltwater
oai-searchbot
omgili
omgilibot
perplexitybot
piplbot
seekr
truliabot
youbot

Rule Path
Disallow /