xth.jp
robots.txt
Robots Exclusion Standard data for xth.jp
Resource Scan
Scan Details
| Site Domain | xth.jp |
| Base Domain | xth.jp |
| Scan Status | Ok |
| Last Scan | 2025-12-24T20:27:28+00:00 |
| Next Scan | 2025-12-31T20:27:28+00:00 |
Last Scan
| Scanned | 2025-12-24T20:27:28+00:00 |
| URL | https://xth.jp/robots.txt |
| Domain IPs | 162.43.94.16 |
| Response IP | 162.43.94.16 |
| Found | Yes |
| Hash | 79f40e86f65a1defaaeda14164449123bb4e67c4f58023d6227d071c56d2388b |
| SimHash | 2811d911f211 |
Groups
baiduspider-render
meta-externalagent
meta-webindexer
cotoyogi
applebot
applebot-extended
ccbot
deepseek v3
gptbot
oai-searchbot
chatgpt-user
copilot
claudebot
claude-web
claude-searchbot
anthropic-ai
cohere-ai
the knowledge ai
scrapy
imagesiftbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /leaflet/ |
| Disallow | /*PageSpeed%3Dnoscript$ |
| Disallow | /mod_pagespeed_beacon/ |
| Disallow | /common/ |
| Disallow | /checkout/ |
| Disallow | /error.html |
| Disallow | /*.js$ |
| Disallow | /*.css$ |
| Disallow | /403.html |
| Disallow | /404.html |
| Disallow | /common/ |
| Disallow | /script/ |
| Disallow | /cron_make_sitemap_for_xth.php |