it.mashable.com
robots.txt

Robots Exclusion Standard data for it.mashable.com

Resource Scan

Scan Details

Site Domain it.mashable.com
Base Domain mashable.com
Scan Status Ok
Last Scan2025-08-19T09:10:42+00:00
Next Scan 2025-09-18T09:10:42+00:00

Last Scan

Scanned2025-08-19T09:10:42+00:00
URL https://it.mashable.com/robots.txt
Domain IPs 23.54.118.38, 23.54.118.40
Response IP 23.32.39.137
Found Yes
Hash b5f9613396f7829869e9206978c463972bbd7d9debe9a49eb1be8779ab55f7aa
SimHash 7c521159cbb6

Groups

*

Rule Path
Disallow /admin/
Disallow /*/admin/
Disallow /se/
Disallow /*.psd$
Disallow /apiproxy/

ai2bot-dolma
anthropic-ai
amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
ddm*
ddm-dcipher/1.0.7
ddm-dcipher*
diffbot
duckassistbot
facebookbot
gptbot
httrack
meta-externalagent
meta-externalfetcher
nutch
oai-searchbot
offline explorer
omgili
perplexity-user
perplexitybot
scrapy
timpibot
youbot

Rule Path
Disallow /

Comments

  • Ziff Davis content is made available for your non-commercial use subject to our
  • Terms of Use here: https://www.ziffdavis.com/terms-of-use
  • Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on
  • this website using automated means is prohibited without written permission from Ziff Davis.
  • Prohibited uses include but are not limited to:
  • (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single
  • Market;
  • (2) development or operation of artificial intelligence or machine learning software or
  • databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;
  • (3) creating data sets containing our content or sharing it with others; and
  • (4) any commercial purposes.
  • Contact licensing@ziffdavis.com for assistance.

Warnings

  • 1 invalid line.