asia.nikkei.com
robots.txt

Robots Exclusion Standard data for asia.nikkei.com

Resource Scan

Scan Details

Site Domain asia.nikkei.com
Base Domain nikkei.com
Scan Status Ok
Last Scan2024-11-06T01:38:43+00:00
Next Scan 2024-11-20T01:38:43+00:00

Last Scan

Scanned2024-11-06T01:38:43+00:00
URL https://asia.nikkei.com/robots.txt
Domain IPs 151.101.131.52, 151.101.195.52, 151.101.3.52, 151.101.67.52, 2a04:4e42:200::820, 2a04:4e42:400::820, 2a04:4e42:600::820, 2a04:4e42::820
Response IP 199.232.47.52
Found Yes
Hash 69307d83236b7dc82ec61f826421d1c2d75c5d073d9b4695a8ee0616a791b724
SimHash 349d9b64c78a

Groups

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /search

yahoo! slurp

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

applebot

Rule Path
Disallow /search

googlebot

Rule Path
Disallow /search
Disallow /member/register/?n_cid=NARAN003
Disallow /member/register/?_ptid
Disallow /member/register?n_cid=NARAN003
Disallow /member/register?_ptid
Disallow /__service

adsbot-google

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

bingbot

Rule Path
Disallow /search

yandexbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

zumbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

spiderbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

dts agent

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

taboolabot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /search
Disallow /__service
Disallow /cms
Disallow /api

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /