Log In Sign Up

xtrend.nikkei.com
robots.txt

Robots Exclusion Standard data for xtrend.nikkei.com

Resource Scan

Scan Details

Site Domain xtrend.nikkei.com
Base Domain nikkei.com
Scan Status Ok
Last Scan2024-09-24T19:31:55+00:00
Next Scan 2024-10-08T19:31:55+00:00

Last Scan

Scanned2024-09-24T19:31:55+00:00
URL https://xtrend.nikkei.com/robots.txt
Domain IPs 52.197.103.112, 54.95.97.7
Response IP 54.95.97.7
Found Yes
Hash cf2134932277329434d79cceb384ff6286847897d4d026f6360e1d558b3aec4d
SimHash fa1da80183a3

Groups

*

Rule Path
Disallow /*leafChildJson
Disallow /atcl/calendar/
Disallow /navi/support/

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

webzio

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://xtrend.nikkei.com/sitemap.xml
Back to top