tech.yahoo.com
robots.txt
Robots Exclusion Standard data for tech.yahoo.com
Resource Scan
Scan Details
| Site Domain | tech.yahoo.com |
| Base Domain | yahoo.com |
| Scan Status | Ok |
| Last Scan | 2025-07-17T00:02:43+00:00 |
| Next Scan | 2025-07-31T00:02:43+00:00 |
Last Scan
| Scanned | 2025-07-17T00:02:43+00:00 |
| URL | https://tech.yahoo.com/robots.txt |
| Domain IPs | 202.165.107.57, 2406:2000:e4:1404::1:2000 |
| Response IP | 202.165.107.57 |
| Found | Yes |
| Hash | ba2b21846f3af906b3ff155917db3530f7e801682e0d87276b23ddab5259ff48 |
| SimHash | 75057b12c080 |
Groups
admantx
alphabot
anthropic-ai
awariorssbot
awariosmartbot
blexbot
buzzbot
bytespider
ccbot
chatgpt-user
claritybot
claude-web
claudebot
cohere-ai
diffbot
facebookbot
friendlycrawler
google-extended
gptbot
huggingface
imagesiftbot
img2dataset
magpie-crawler
meltwater
neevabot
news-please
newsnow
nutch
omgili
omgilibot
http://panscient.com
perplexity-ai
perplexitybot
petalbot
piplbot
http://scoop.it
scrapy
seekr
sentibot
seznambot
turnitinbot
youbot
zumbot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://tech.yahoo.com/sitemap.xml |
| sitemap | https://tech.yahoo.com/news.sitemap.xml |