it.mashable.com
robots.txt
Robots Exclusion Standard data for it.mashable.com
Resource Scan
Scan Details
Site Domain | it.mashable.com |
Base Domain | mashable.com |
Scan Status | Ok |
Last Scan | 2025-08-19T09:10:42+00:00 |
Next Scan | 2025-09-18T09:10:42+00:00 |
Last Scan
Scanned | 2025-08-19T09:10:42+00:00 |
URL | https://it.mashable.com/robots.txt |
Domain IPs | 23.54.118.38, 23.54.118.40 |
Response IP | 23.32.39.137 |
Found | Yes |
Hash | b5f9613396f7829869e9206978c463972bbd7d9debe9a49eb1be8779ab55f7aa |
SimHash | 7c521159cbb6 |
Groups
*
Rule | Path |
---|---|
Disallow | /admin/ |
Disallow | /*/admin/ |
Disallow | /se/ |
Disallow | /*.psd$ |
Disallow | /apiproxy/ |
ai2bot-dolma
anthropic-ai
amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
ddm*
ddm-dcipher/1.0.7
ddm-dcipher*
diffbot
duckassistbot
facebookbot
gptbot
httrack
meta-externalagent
meta-externalfetcher
nutch
oai-searchbot
offline explorer
omgili
perplexity-user
perplexitybot
scrapy
timpibot
youbot
Rule | Path |
---|---|
Disallow | / |
Warnings
- 1 invalid line.
Comments